Final Report Summary - IMARS (iMars: Analysis of Mars multi-resolution images using auto-coregistration, data mining and crowd source techniques)
Executive Summary:
Executive Summary
The iMars project focused on developing tools and value-added datasets to massively increase the exploitation of space-based data from NASA and ESA Mars mission imaging and 3D data beyond the PI teams. iMars has added significant value by creating more complete and fused 3D models of the surface from multi-resolution co-registered stereo all co-registered to a global reference system derived from laser altimetry and has shown how these 3D models can be employed to create a set of co-registered imaging data through time, permitting a much more comprehensive interpretation of the Martian surface to be made.
Emphasis was placed on the co-registration of multiple datasets from different space agencies and orbiting platforms around Mars and their synergistic use to discover what surface changes have occurred since NASA's Viking Orbiter spacecraft first went into orbit around Mars forty years ago.
iMars brought together the best expertise in Europe for the processing of Martian orbital data within a single environment for handling, visualising and interpreting these data. The ESA Mars Express High Resolution Camera (HRSC) provided the 3D mapping products used as base data (for around 50% of the surface), where possible. When CTX stereo products are also available over the same areas as HRSC (for around 20% of the surface), then the CTX products can be co-registered with HRSC and CTX 3D mapping products can be employed as the base data for higher resolution images such as MOC-NA and HiRISE. A Co-registered Ames Stereo Pipeline using Gotcha Optimisation (CASP-GO) was developed for large-scale production of CTX 3D mapping products and small area production of HiRISE products. Some 5,300 CTX stereo products have been processed using cloud computing provided by Microsoft Azure® covering about 20% of the Martian surface. In iMars, standards were set for the production and dissemination of HRSC mosaiced products, which are easier to utilise for co-registration than individual strips and have better internal geometry. A fully automated Auto-Coregistration and Orthorectification (ACRO) system was developed to operate on a linux cluster without any manual intervention. Around 15,000 NASA images (out of the ≈400,000 acquired with resolution of ≤100m) were processed using the ACRO system covering around 4% of the surface. New HRSC 3D mapping products were produced for the South Polar Residual Cap area and the unfunded (by iMars) PhD student is working on applying the same approach to the North Polar cap in collaboration with another iMars partner. ACRO products have therefore been processed as well as multi-resolution DTMs from CTX and HiRISE. All the 3D mapping products have been analysed qualitatively by visual inspection and the ACRO processing includes calculating internal quality metrics, which is used to flag bad products.
An automated data mining algorithm was developed to find scene fragments where single or multiple instances of change are detected. This used supervised classification initially with planetary science labelled inputs but will in the near future employ mass public participation from a shortly-to-be-launched citizen science programme under the auspices of Zooniverse. A great deal of effort was placed on determining the optimum Human Factors for incentivising and motivating such participants. Each additional set of a minimum of 10 identifications and notations will be further employed to improve the classification (standing it is believed at around 50%) and the data mining applied to new areas by non-EU funded students at Masters and PhD level. The key to public dissemination is the iMars webGIS system which has been developed to allow experts and members of the public to examine different parts of the planet for changes as well as perform geomorphological and geological research. Dissemination of the underlying products will take place through ESA PSA (for HRSC products) and NASA PDS (for NASA ACRO and CASP-GO 3D mapping products).
The iMars base data can be used by the ESA ExoMars Trace Gas Orbiter 2016 and subsequent ESA missions to provide the necessary inputs for the selection of a future landing site for the ESA ExoMars 2020 rover and for any Mars Sample Return missions in the 2020s.
It will greatly extend the use of the archived data by providing mapped and co-registered images.
Project Context and Objectives:
Context and Objectives
Almost fifty years have elapsed since the NASA Mariner 4 first took the first pictures of the Martian surface. Over that time, the resolution and quality of these images has improved from tens of kilometres down to 25 cm. Over the intervening ≈50 years, many areas on Mars have been repeatedly imaged as each new science team has “re-invented the wheel” and gone back to the same places highlighted by these early images. Historically, the driver for these scientific studies has been the “hunt for water” but more recently, there is an increasing interest in different dynamic phenomena associated with motion of dust and possibly different permafrost phenomena. The revolution in planetary surface observations, especially in 3D imaging of surface shape, has led to the ability to be able to overlay different time epochs back to the mid 1970’s, to examine time-varying changes, such as the recent discovery on Mars of mass (e.g. boulder) movement, tracking inter-year seasonal changes and looking for fresh craters from meteoritic strikes. As our exploration of surface changes proceeded we determined that the polar regions had some of the most significant surface changes and so a great deal of emphasis was changed to focus on these regions.
In iMars, we wanted to mitigate the deleterious effects of increasing knowledge and accuracy of where the spacecraft was located in its orbit and where the cameras were pointed to allow fully automated computer algorithms to eliminate the impact of these errors and provide co-registered image products so we can playback any area on Mars, which had repeat coverage. Of course, in many cases no change can be observed, however surprisingly the surface is much more dynamic than at first thought particularly with regard to very dynamic phenomena like dust devils. When the technology developed by iMars is rolled out across the planet in future it is very likely that significant new discoveries will be made. For polar regions, the number of repeats is significant as well as for regions such as the landing sites, Vallis Marineris, the Tharsis volcanic region and other regions repeatedly imaged on the boundary of the so-called dichotomy. Little, if any, information is available on the variability of the Martian surface, although the annual and seasonal changes associated with the polar cap expansion and contraction and the rapid spread of dust storms has been relatively well monitored in the past. However, the Martian surface is undergoing constant changes from our observations of just 3% of the surface so we do not yet know what we can fully expect to find.
Objectives (taken from the original DoW):
• To characterize the nature and characteristics of HRSC sensor orientation variations within Mars orbit.
• To assess suitable techniques for extracting height control information from MOLA altimetry.
• To compile a contiguous HRSC DTM data product by adjustment and integration of stereo-image coverage on multiple overlapping image tracks on global scale and assess the performance achieved.
• To show the impact of multi-orbit integration on an improvement in DTM mosaicing.
• To integrate the HRSC DTM data product generated in Task 3.2 into the global project data base.
• To mosaic the resultant DTMs with existing MOLA profile data to ensure the best possible global DTM. Surface matching to be employed rather than rigorous block adjustment.
• To generate the most complete CTX stereo-derived DTMs and ORIs for much of the Martian surface and employ the automated co-registration software to ensure that these images are co-registered to the HRSC ORIs. To generate a set of ORIs for single view CTX images using the best DTM.
• To generate the most complete HiRISE stereo-derived DTMs and ORIs feasible and employ the automated co-registration software to ensure good co-registration with CTX.
• To apply the automated co-registration software using the HRSC-CTX-HiRISE ORIs to all prior NASA Mars imagery from Mars Global Surveyor, Mars Odyssey and Viking Orbiter back to 1976. The results of each feature-pair match will be fed into updating the SPICE kernels for each historical image and subsequent orthorectification using the global DTM.
• To ensure that all the resultant images are entered into the web-GIS database and can be accessed directly within existing GIS such as ARC® and GRASS.
• To generate a Design Document (DD) consisting of functional specifications and implementation approaches for a planetary Mars-based webGIS (hardware), DBMS and GIS service specifications, user access specifications, and specifications for integration of software modules defined through a common architecture API (software).
• To setup a development and testing/validation webGIS core-system environment.
• To implement modules developed by consortium.
• To setup an environment (as specified in the DD) for open as well as credential-based access by researchers and general public.
• To provide access and means for validation and stress-testing of the webGIS and, by that, demonstrate operability.
• To investigate the use of co-registered datasets and available matching tools for change detection.
• To explore different change detection methods applied to time series images of the Martian surface.
• To choose the best possible method to flag images where change has occurred including time-lapse sequences to assess whether such a sequence is a true change or just an error of commission.
• To provide data to WP7 for validation using crowd sourcing.
• Review human factors research on human computation & automation.
• Test HF theory in the domain of Martian geomorphological feature recognition.
• Build a MarsZoo project to produce data on a range of features, taking into account the results of HF testing.
• Validate MarsZoo results with scientific users.
• Communicate “best practice” results & guidelines.
• To coordinate dissemination activities among all project partners and ensure visibility of project aims and results to science community and general public.
• To develop strategies for further exploitation of the project results and identify opportunities for follow-up projects.
• To implement efficient communication with scientific community and general public for enhancing and promoting the use and benefit of iMARS tools and data.
• To employ the NASA Regional Planetary Imaging Facilities (RPIFs) to promote the products and tools developed by iMars.
• Ensure that adequate technical and administrative management procedures are established and implemented.
• Report on project status and achievements.
• Implement a policy leading to the minimisation of project risks via timely identification of problem areas and their resolution.
• Perform timely and correct financial project management.
Project Results:
Description of Main Results
The iMars project focused on developing tools and value-added datasets to massively increase the exploitation of space-based data from NASA and ESA mission imaging and 3D data beyond the PI teams. iMars has successfully created substantially more complete and fused 3D models of the surface from combined stereo and laser altimetry and used these 3D models to create a set of co-registered imaging data through time, permitting a much more comprehensive interpretation of the Martian surface to be made by scientists in the future. Emphasis has been placed on co-registration of multiple datasets from different space agencies and orbiting platforms around Mars and their synergistic use to discover what surface changes have occurred since NASA’s Viking Orbiter spacecraft in the mid-1970’s. iMars has brought together the best expertise in Europe for the processing of Martian orbital data within a single environment for handling, visualising and interpreting these data. The ESA Mars Express High Resolution Camera (HRSC) has been used as base data, where possible as its fundamental image resolution of 12m and DTM resolution (from 50-150m) makes it ideally suited to processing NASA imagery of resolution from 6m-100m.
The iMars datasets/products can be used to provide the necessary inputs for any Mars Sample Return missions in the 2020s. It greatly extends the potential use of archived data by providing mapped and co-registered images. The resultant time-stamped imagery has been interfaced to automated data mining analysis software based on techniques developed for Earth surveillance. We have also built on the huge momentum, developed in the Zoouniverse system by building a “Mars in Motion” project for mass public participation in the feature mapping of Mars. Successful co-operation with US colleagues through the Technical Advisory board at annual project meetings, as well as participation from European scientists through several workshops have brought the project to a greater level of scientific interest and a public event at the end of the project have helped to promote the project to a much wider audience.
WP1 Software for Automated DTM/ORI extraction from CTX/HiRISE data
In WP1, UoS and EPFL developed a stereo processing system while UCL assessed existing processing chains from NASA and USGS and compared the results from UoS and EPFL with these for both accuracy, completeness and speed. Results were then compared against each other and with the HRSC mapped products to find a suitable processing system for large-scale processing.
UoS modified an automated stereo processing pipeline developed by Kim and Muller (2009) for a more generic planetary stereo processing into an automated DTM pipeline and advanced sensor-modelling system. Consequently due to a successful “coarse to fine” control strategy, stable horizontal and vertical photogrammetric accuracy of the resultant iMars DTMs was achieved when compared with a MOLA DTM. The UoS algorithms were successfully applied to produce 12m CTX and 0.75m HiRISE grid spacing Digital Terrain Models (DTMs) and ortho-rectified images (ORIs) after application to CTX and HiRISE stereo images.
The major differences to the original algorithm published by Kim and Muller (2009) for the UoS stereo processor are:
1) Employing the ISIS “cub” level 1 image matching software.
2) Requirement for only a few 3rd party software libraries. Minimising unnecessary components aiming to create a certain quality DTMs with only essential modules.
3) Minimising parameter dependency during the stereo matching process.
4) Final system consists of C++ execution files and a Python wrapper.
5) All main execution files were designed with openMP functionality for high speed parallel processing on the MSSL linux cluster.
The UoS software delivered to UCL consisted of the following components
1) Conversion routines to permit a generic sensor model to be generated from interior & exterior orientation parameters.
2) Efficient stereo image matcher which can manipulate both texture-less and high slope topographies together
3) Bundle block adjustment along with space resection software routines
4) Space intersection routines
5) Other auxiliary routines to handle topographic product
EPFL had previously developed an additional processing pipeline for high resolution DTMs. It employs stereo matching methods and approaches utilised for MOC stereo data (Ivanov, 2003; Ivanov & Lorre, 2002) in conjunction with radiometric and geometric image processing in ISIS3. This technique is capable of deriving tie-point co-registration at sub-pixel precision and has proven itself when used for Pathfinder and Mars Exploration Rover, MER operations (Lavoie et al., 1999).
The basis for image correlation is the Gruen algorithm (Gruen, 1985), which was originally employed in the amoeba optimization method (Nedler and Mead, 1965). The method is an iterative approach. Given template image and search area, it searches for the best match in the search area. This search can take a considerable amount of time and therefore this part of the code is the prime candidate for implementation on parallel architecture.
The next step is performed using the ISIS3 Application program Interface (API). The ISIS3 API is used to derive ray information for ray-ray intersections for pixels identified by image correlation. Once ray intersections have been calculated they give precise XYZ information in a Mars Reference frame. XYZ values are then converted into triplets of (Latitude, Longitude, Radius). The final product is created by triangulation and gridding of this dataset. UoS utilise Generic Mapping Tools (GMT) to produce regularly spaced grids, using interpolation in places where we have found no correlation. Filtering bad correlation points is an important step for the final quality of the DEM and this step still has to be tuned for each stereo-pair in order to achieve the best possible DTM quality.
UCL applied the NASA AMES stereo pipeline (Moratto et al., 2010) “as is” as well as the USGS stereo processing system based on the use of ISIS software for creation of images from the original level-1 HiRISE separate pushbroom files and the COTS SOCET® system for the DTM and ISI for the ORI production. The UoS software was ported to UCL but good results were not achieved.
The following products were produced for the WP1 assessment by EPFL, UoS and UCL:
CTX and HiRISE DTM and ORIs co-registered to HRSC DTM & ORI for the selected data sets for the sites covering three NASA Mars rover systems: MSL-Curiosity, MER-A Spirit and MER-B Opportunity sites with the following specifications:
• Grid spacing (hereafter called resolution) for CTX DTMs (20m) and CTX ORIs (6m);
• Resolution for HiRISE DTMs (0.75m) and ORIs (O.25m);
• Output data format: geotiff with separate world files.
A pairwise comparison of any set of algorithms as described above was performed for the same regions on Mars and compared against corresponding HRSC DTMs of lower resolution.
For this purpose we chose to perform an evaluation of CTX images of the area surrounding the three most observed rover sites. The supporting DTMs and images (for selecting geodetic GCPs) were taken from the HRSC products overlapping with the CTX images. Subsequently, some HiRISE stereo-pairs overlapping with the selected CTX images were also processed using SOCET® and the NASA-ASP system for comparison purposes.
The average and standard deviation of the differences between the HRSC and ASP, SS, and UoS DTMs for CTX was +1.4 ± 84.2m −2.1 ± 84.4m and −2.7 ± 84.9m respectively. The same differences for the HiRISE instrument were −13.3 ± 19.7m +4.2± 19.7m and +2.3 ± 37.2m. The large dispersion of the differences is due to a larger number of surface features over a larger area for CTX and a smaller number of features for a smaller area covered by the HiRISE instrument. The results were considered to require further improvement in global consistency, completeness and robustness in WP4.
In a further validation part of WP1, DLR found large height offsets between WP1 DTM results and MOLA/HRSC and could relate this to the height datum definition in the USGS software system (ISIS). DLR DTM validation revealed a miss-alignment of the default MOLA data set of ISIS with respect to the reference PDS data set. Further analysis on this issue revealed fundamental inconsistencies in the GDAL and ISIS routines for HRSC import as well. Data conversion procedures strictly applying the written definitions for the respective software systems and data products have been set up in order to be able to validate the CTX/HiRISE data products. The latest version of GDAL took these reported issues into account to address future issues. D1.1 and D1.2 were produced for this WP.
WP2 Software for Auto-coregistration of NASA images to HRSC ORI/DTMs
UCL developed an automated algorithm for the co-registration of feature points extracted from overlapping NASA images at level-1 called EDR (Experimental Data Records) to ESA-DLR-FUB HRSC mapped data. This automated co-registration and orthorectification (ACRO) system has a number of novel features, including the use of SIFT features, coupled decomposition for sub-image creation for massive >1Gb sized images, detection of sub-pixel features in HRSC and corresponding correlated features prior and future non-HRSC (i.e. NASA) images and creation of a set of latitude/longitude/height points in HRSC and overlapping NASA images which can be employed in future to update the SPICE kernels for these NASA images as well as be employed as geodetic control points for HiRISE, CTX, CaSSiS and serendipitous stereo from other NASA sensors.
Cameras Years Resolution (m) No. images Data
Viking Orbiter 1 1976-1980 8-1800 ~32 000 29G
Viking Orbiter 2 1976-1978 8-1800 ~15 000 29G
MGS MOC-NA 1997-2006 1.5-12 97 097 ~350G
MO THERMIS 2002- 18-36 ~205 000 ~6T
MeX HRSC 2004- 12.5-25 ~5 000 (nadir) ~3T
MRO CTX 2006- 6 ~79 000 ~12T
MRO HiRISE 2006- 0.25-0.5 ~90 000 ~120T
Table 1: Summary of High-resolution (≤100m) Mars image data (aside form VO). Note that HiRISE acquires the same data volume every day that 2 VO missions acquired over 4 years.
Table 1 summarizes the numbers of orbiter images and associated data volume acquired to date. This large quantity of data was employed to generate repeat coverage maps of the Martian surface [see Figure 1: High-Resolution Mars REPEAT Coverage taken from Sidiropoulos & Muller 2015], thus making it possible to assess where surface changes may be able to be tracked over the planet. Table 1 shows that, even if images with resolution finer than 20 metres per pixel are only taken into account, change detection is theoretically possible over a significant part of the Martian surface. For example, the area mapped more than once during (Northern Hemisphere) spring is larger than the entire area of Asia, while the area mapped at least thrice in any season is on the order of an Earth’s continental area.
The UCL ACRO software generates orthorectified images of the Martian surface from orbital data of NASA missions to Mars (VO, MOC-NA, THEMIS, CTX and HiRISE instruments) co-registered to HRSC orthorectified images and orthorectified using HRSC DTMs. See Sidiropoulos & Muller (2017) and deliverables D2.1 & D2.2 for further details of the algorithm and software.
In parallel, UoS conducted co-registration of their target area images and DTM products over HRSC employing their in-house co-registration routine. The main purpose of UoS efforts are the establishment of ground control point setting for epipolar rectification and geodetic control for the WP4 stereo processing. In order to achieve fine resolution and high accuracy geodetic control with less than few horizontal pixels and 10m vertical deviations, a robust geodetic control method was implemented combining image-to-image correlations. The algorithm is based on a combination of machine vision approaches consisting of feature point establishment using Scale Invariant Feature Transformation (SIFT), a variation of an optical flow algorithm, least squares correlated refinement and pyramidal reorganization employing affine transformation. UoS also implemented an image pointing adjustment method using rigorous and non-rigorous sensor models, which is then applied to co-registration of stereo photogrammetric products. UoS studied and tested involved procedures using their in-house software and a combination of some subroutines from the open source USGS ISIS.
WP3 Bundle block-adjusted DTM mosaic from HRSC
The iMars project is developing a user platform for Mars surface change science, consisting of a collection of co-registered and terrain corrected data products from Mars orbital imaging data sets, and specific tools for producing, exploring and analysing change shown in these (data mining, webGIS, crowd sourcing). For GIS analysis, and in particular for change detection, co-registration of all involved datasets to a common geometric reference is essential. The primary reference data sets used by iMars for this purpose are the digital terrain models and orthoimages derived from HRSC map products. Therefore availability of these data products together with a detailed assessment of their geometric characteristics has been a major cornerstone within the project.
The technical reasons for this are intrinsic difficulties associated with co-registering images to a reference dataset represented by an elevation model such as the NASA Mars Orbiter Laser Altimeter experiment (MOLA), such that for the case of the highest resolution data sets such as data of the MOC narrow angle imaging device of Mars Global Surveyor or the HiRISE camera of Mars Reconnaissance Orbiter (MRO), a geometric relationship with the MOLA reference usually cannot be established directly due to the huge differences in spatial resolution. Conversely, HRSC, as a stereo imaging sensor and with imaging at a resolution of 12 m, provides data that can demonstrably be precisely registered to MOLA heights by means of photogrammetric adjustment, so that HRSC terrain-corrected images [see Figure 2: Colour-coded shaded relief maps of the HRSC multi-orbit DTMs for Mars Quadrangles MC-11-E (Oxia Palus, East) and MC-11-W (Oxia Palus, West). E-W extent is about 2663 km (at the equator), N-S extent about 1780 km, grid spacing of 50 m.] can in turn be used as a basis for co-registering other image data.
One of the major objectives of WP3 consisted in providing the necessary specifications concerning HRSC multi-orbit data products as a new base data set, in particular their use as a reference dataset. In addition, a review of relevant methodological background is provided as well as a summary of specific investigations in the areas of height control and modelling of image orientation that were intended to provide useful guidance for optimising DTM production. The dataset specifications were discussed in the context of previous topographic Mars data products and current usage trends for such data products. The compiled set of product specifications is available to the Mars Express HRSC team for adaption to the systematic generation of HRSC multi-orbit products.
The suggested parameters specifying the multi orbit data products were obtained using processing tests concerning the representation of detail in the DEM, the internal geometric consistency between strips, and the consistency with external reference heights.
Furthermore, we analysed the original HRSC mission data record with respect to important parameters such as image resolution, we reviewed and analysed existing results of single-strip product generation by the HRSC Team, performed image mosaicing tests that were evaluated in terms of brightness homogeneity and the effectiveness of image edge elimination, and, finally, reviewed relevant PDS standards. The analysis of the methods for height control was based on processing results obtained using two different tools, bundle adjustment and a procedure for trend surface analysis and correction. The results provided different implications for the suggested set-up of the product generation, most importantly a recommendation to apply bundle adjustment as the general 3D adjustment method in combination with a final trend surface correction for height. The analysis of Mars Express orientation data properties was based on analysis of existing results from HRSC single-strip processing, comparison with available high sample rate orientation data from ESA, and star observations by the SRC camera. The results showed that a tie-point sampling rate of 4 seconds is generally required to cover typical MEX spacecraft oscillations, and 2 seconds are recommended considering the finite precision of tie-point measurement.
The second main focus of WP3 was devoted to the geometric validation of HRSC multi-orbit DTM products, which is a very relevant aspect concerning the further use of such data as a geometric reference dataset. Two DTM prototype datasets were used for validation (MC-11E and MC-11W). The geometric quality of the results has been assessed with respect to internal parameters such as the mean 3D intersection error as well as external data, i.e. MOLA altimeter heights. The results were rated very encouraging because they confirmed that the already high and well-controlled internal precision of the single-strip 3D models produced and released to the PDS/PSA archives for several years are preserved throughout the new integrated bundle block adjustment process. We also investigated the quality of co-registration of adjacent HRSC strips before and after block adjustment and could show that residual offsets are reduced by a factor of 3 to 4 for the horizontal coordinates, or to about one pixel for the nadir images, which allows the production of panchromatic orthoimage mosaics with the highest available resolution. A comparison of the results of our new approach to multi-orbit processing with simple mosaics of local DTM patches or strips demonstrated that typical obstacles to the successful application of the latter can be efficiently avoided by the new approach. This refers to avoidance of edge artefacts, related to weakly constrained interpolation close to DTM borders, masking of higher resolution datasets by lower resolution datasets, and increased coverage by filling of data gaps present in some of the datasets.
WP4 Global DTM/ORI production & validation
The HRSC data products in the iMars data base include more than 1402 single strip PDS HRSC DTMs and orthoimages produced by DLR, as well as the multi-orbit products for the MC-11E and MC-11W tiles used for validation in WP3 which themselves comprise some 200 single-strips. At a recent HRSC CoI meeting held just after the HRSC CoI meeting, DLR announced that out of 16,788 orbits performed by the ESA Mars Express mission, some 5,000 contained HRSC data with 75.9% having nadir images with <20m resolution and L4 products being available up to orbit 8,000 within the HRSC CoI team and shortly through PSA/PDS to the scientific community. It appears that around one quarter have been processed to date with DLR recently re-commencing the production of single-strip bundle adjusted L4 products at a rate of several tens per month.
DLR produced Lambertian BRDF-corrected, block adjusted orthoimages in 5 filters over the entire MC-11W and MC-11E areas. These orthoimages were brightness-adjusted and mosaiced using a global set of reflectance standards generated from the Mars Global Surveyor, TES instrument. In addition, all the data from Mars Express orbits between 2004 and 2016 covering the MC-11E and MC-11W areas were processed to Level-4 single strip data products (DTMs and orthoimages) using the new bundle-block-adjusted orientation data. These represent a preliminary version of new, updated Level-4 products to be derived from bundle block solutions in the future. For this reason the data sets are provided for internal use by the consortium in the frame of iMars and will be replaced in the near future by an official version released to PSA/PDS.
In addition, 39 HRSC orbits were processed by UCL with EXTORI files from FUB to produce complete coverage of the South Polar Residual Cap both with 3D Digital Terrain Model (DTM) and a set of orthoRectified images (ORIs) shown in Figure 3 [Base SPRC DTMs produced by UCL and FUB], and, Figure 4 [Orthorectified Nadir images using base SPRC DTMs]. See D4.1 report for further details.
In WP4, a fully automated multi-resolution DTM processing chain was developed by UCL for NASA CTX and HiRISE stereo-pairs, called the Co-registration ASP-Gotcha Optimised (CASP-GO), based on the open source NASA Ames Stereo Pipeline (ASP) (Moratto et al., 2010; Broxton et al., 2008), tie-point based multi-resolution image co-registration (Sidiropoulos & Muller, 2015), and Gotcha (Shin & Muller, 2012) sub-pixel refinement method. The implemented system guarantees global geo-referencing compliance with respect to High Resolution Stereo Colour imaging (HRSC), and hence to the Mars Orbiter Laser Altimeter (MOLA), providing refined stereo matching completeness and accuracy from the ASP normalised cross-correlation.
The CASP-GO pipeline for both sensors is shown in Figure 5a: Flow diagram of UCL-Ames CASP-GO processing chain for CTX, and, Figure 5b: Flow diagram of UCL-Ames CASP-GO processing chain for HiRISE. Apart from the ASP pre-processing, cross-correlation matching, triangulation, and DTM/ORI generation, five additional workflows are introduced to further improve the ASP results. These include (a) a fast Maximum likelihood sub-pixel refinement method to build a floating-point initial disparity map; (b) an outlier rejection and erosion scheme to define and eliminate mis-matches; (c) an ALSC and region growing (Gotcha) based refinement and densification method to refine the disparity value and match un-matched and/or mis-matched area; (d) co-kriging grid-point interpolation to generate the final DTM as well as height uncertainties for each DTM point; (e) ORI co-registration w.r.t. HRSC.
Based upon the “cleaned-up” sub-pixel disparity map, an Adaptive Least Squares Correlation (ALSC) refinement is performed on all the remaining disparity values iteratively. These refined disparity values are used as seed points for Gotcha (Grün-Otto-Chau) densification (Shin & Muller, 2012). The Gotcha matcher is based on ALSC and region growing. It is very robust and accurate, but rather slow for large-scale image matching since it tries to match every point iteratively and re-sort all seed points according to a “best first” strategy when a new patch is matched. However, given sufficient number of sub-pixel disparities that pass the outlier rejection schemes, small difficult regions can be matched with Gotcha accurately. For example, the geometrical distortion generated by different viewing angles can be addressed with Gotcha by modifying the shape of the ALSC window iteratively, albeit currently only with a parallelepiped using an affine transformation for the distortion model.
With Gotcha densification, we can achieve improved completeness for the final DTM without significantly smoothing out sharp features from a large matching kernel. Generally, a larger ALSC window and higher maximum eigenvalue yields better completeness in disparity and hence for any resultant DTM. The similarity values from the Gotcha matcher are then used together with camera model intersection error and co-kriging parameters to produce DTM uncertainty values per gridpoint.
In CASP-GO, UCL added a Mutual Shape Adapted Scale Invariant Feature Transform (MSA-SIFT) based co-registration workflow to the final ORI and DTM product. Technical details of MSA-SIFT and the evaluation on ORI/DTM co-registration results for MER and MSL are described in (Tao & Muller, 2015). In this ORI co-registration and DTM adjustment work, we take HRSC ORI as the reference image for CTX ORI co-registration and subsequent shift of the corresponding CTX DTM according to the CTX ORI to HRSC ORI transformation. For the HiRISE co-registration, we take already co-registered CTX ORI (to HRSC ORI) as the reference image for HiRISE ORI co-registration and subsequent transformation of the HiRISE DTM to ensure multiscale congruency.
UCL mirrored the HRSC, CTX, HiRISE PDS data volumes from JPL in a local shared storage system in order to speed up the production process with an option such that if data is unreachable it can be read from the original source again. At UCL-MSSL, the developed software is installed in a shared directory, which is accessible from 14 Linux processing blades (10 with 16 cores and 48GB RAM; 4 with 24 cores and 96GB RAM). Jobs are controlled via a local desktop machine and distributed to the 14 processing blades with multiple sessions of multi-threaded processing. Processed results are stored in several 1TB RAID storage disk partitions and logged back to the local controlling desktop. Failed jobs can be examined through detailed log files and could be reprocessed automatically with different processing parameters in the future.
In the meantime, UCL was successfully awarded with free access to $20,000 of computing resources from Microsoft® Azure Cloud for Research. UCL worked on the virtual machine set-up, software integration, and test processing on the Microsoft® Azure cloud computing since early 06/2016 and started batch processing of 1540 CTX stereo pairs (the original published list of CTX stereo-pairs from the NASA Ames group published online in 2012) in 07/2016 [see Figure 6: Location of all 1540 18m CTX DTMs processed by the end of 2016. Where no level-4 HRSC DTM exists (red areas), the CTX products are not co-registered to a global co-ordinate system] which completed in mid 12/2016. UCL was later awarded an additional $20,000 computing resources from Microsoft® Azure Cloud for Research in early 01/2017 for finishing up the rest of the ≈4,000 CTX stereo pairs [see Figure 7: Locations of all of the 18m CTX DTMs processed by the Microsoft Azure® including overlaps with those shown above. It is expected that this larger dataset will be completed by mid to end April 2017 after project ends], some of which overlap with the previous set. This new set was derived using a novel stereo intersection algorithm based on the one described by Sidiropoulos & Muller (2015). In all cases only stereo-pairs with 100% overlap were selected.
The CASP-GO processing chain has been tested/applied to stereo Mars Reconnaissance Orbiter (MRO) Context Camera (CTX) imagery (6m) over the Mars Exploration Rover (MER-A, B), Mars Science Laboratory (MSL) and a large area mosaic over the US Geological Survey’s MC11-E/W area (~100 stereo pairs) and then applied to the production of planet-wide DTMs and ORIs from CTX (~1700+3800 stereo pairs) and is being applied MRO High Resolution Image Science Experiment (HiRISE) 25cm NASA images (~11 stereo pairs).
The overall statistics for CTX are 1915 pairs have HRSC out of all of the total of ≈5,300. Out of 1540, 882 pairs don't have HRSC. Out of 3963, 2491 pairs don't have HRSC. However, for the Polar Regions, there is now a complete set of level-4 data so for [see Figure 8: North Pole (a & b) and South Pole (c & d) coverage of CTX in the original 1540 set (left column) and 4000 set (right column). N.B. Please note different Colour key meaning here], these figures are slightly misleading. A large percentage of the processed DTMs have been assessed using a 5 star rating scheme by UCL, i.e. 1 – failed, 2 – major problem, 3 – minor problem, 4 – good quality, 5 – very good quality. 503 out of 620 DTMs have been rated as 3+, i.e. 316 good and very good, 187 have minor problem. Only 31 out of 620 failed all due to bad input images (≈5%).
A geometric quality assessment for the CTX DTMs produced within iMars was performed over 3 test sites (the landing sites of the MER-A, MER-B, and MSL rovers) by DLR. The validation methods included proven techniques for visual assessments based on color-coded height displays, shaded relief maps, and contour lines. In addition, quantitative measures using height statistics, probability of crater detection (to assess effective resolution), and comparison with similar results for existing DTM datasets were applied. A similar set of validation techniques was applied to the HiRISE DTM results over the same three test sites (MER-A, MER-B, MSL landing sites). See D4.5
HiRISE images are 10 to 15 times larger than corresponding CTX images (see further details in D4.3) and contain much richer fine scale features at 25cm that may vary between stereo images. The HiRISE EDR products also contain systematic noise such as strip noise and gain variations. Based on the results from the experimental sites of MER-A, MER-B and MSL, several modifications were made to the original ASP DTM processing chain in order to produce best results for HiRISE images.
A large number of HiRISE DTMs are still being processed in the AWS® cloud computing VMs. These include 441 stereo pairs for the areas that have 5 or more repeat HiRISE observations. This is still on-going due to the large computing resource required and limitations from the free AWS® credit. Once they are completed, the 441 HiRISE DTMs will be integrated into the iMars webGIS together with the existing UoA processed HiRISE DTMs. The global coverage of the UCL HiRISE DTMs that are being processed and the UoA HiRISE DTMs are shown in Figure 9 [Global coverage of the UCL HiRISE DTMs that are being processed and the existing UoA HiRISE DTMs]. Figure 9a shows Global coverage of the UCL HiRISE DTMs being processed and the UoA HiRISE DTMs available to date in Equirectangular projection system. Figure 9b shows the North Pole and Figure 9c shows SPRC, both in Polar Stereographic projection. Purple for the SPRC have existing CTX DTMs].
In parallel, UoS re-implemented WP1 software into a more scientific oriented form especially considering geodetic control and point density, which are essential properties for geological/geomorphological analyses. Together with the UoS stereo software and manual parameter setting, CTX/HiRISE DTMs over scientifically interested areas have been processed and delivered. UoS conducted their own validation work for some CTX and HiRISE products over a few places using mainly MOLA track profiles. This assessment procedure gave confidence for further scientific investigation using UoS DTMs and orthoimages.
Target areas for the application of UoS stereo processors were established in two categories. The processed data sets including DTM and orthoimages over the following areas (> 20 sites):
• Area oriented targets over Naktong valley, Chinju crater, Naju crater
• Feature oriented target areas such as Lobate Debris Apron (LDA) over Euripus Mons, fluvial channel over Baharm and Elysium, tectonic feature around Claritas fossae, Inter-crater features over Elysium, Mojave and Jazero [see Figure 10: Target area locations of CTX and HiRISE DTMs from the UoS stereo processor].
UoS showed that several scientific studies employing UoS processed DTMs are actively undergoing and will be continued in future.
The other main task of WP4 is about the co-registration and orthorectification of NASA imagery to the HRSC baseline, a task that demands not only the development of a core efficient algorithm that can achieve a fast and reliable co-registration of high-resolution multi-instrument planetary images to a common baseline but also a number of variations that overcome the apparently random and incomplete nature of the current high-resolution Mars imagery. These variations were a task that required substantial human resources to be applied. This created a bottleneck in the processing of the global imagery, which while being under way is not expected to be completed before the end of the project. However, a substantial number of images will be processed, and the pipelines for extending this to the rest of Mars orbital imagery are just a matter of computer and human resources, especially since the developed algorithm is fully automatic and requires minimum human resources. Note that UCL has already secured 20,000 core hours of funding from Microsoft Azure® which may be used to process the rest of Mars orbital high-resolution imagery, before January 2018.
Moreover, even though the developed algorithm achieved state-of-the-art resilience to resolution differences between the input image and the baseline, there is a limit on the resolution differences for co-registration in order for it to have any practical meaning. For example, a HiRISE image has a resolution of 25cm/pixel, is 50 times finer than HRSC resolution. This means that a 50x50 HiRISE patch corresponds to a single HRSC pixel. The matching of images with such large resolution difference would be possible if (according to Shannon’s sampling theorem) the spatial frequency of the terrain is such that the sampling difference wouldn’t change the signal (i.e. the image). In practical terms, this would mean features of size no less than 25 metres (since 12.5 m/pixel is HRSC resolution). On Mars, the surface is very rarely so “simple”, since high-resolution imagery both from orbit and from rovers have revealed features of great detail, on scales of centimetres or even millimetres. Under this circumstances, co-registration of images with so much resolution difference is a very challenging task, especially when following the design principle to build a fully automatic pipeline that can process large volumes of data without requiring any parameter tuning. The systematic, batch-mode, co-registration of products with such high-resolution should wait for a 3D baseline of higher resolution (between HiRISE and HRSC), before they can be added to the dataset of Mars geometrical aligned image dataset. CTX is being used for this purpose in future.
Six different variations of ACRO pipeline were developed and applied and are described in D4.4. A composite of 2 CTX ACRO’d images with HRSC is shown in Figure 11: A mosaic of 2 CTX images and an HRSC ORI. The CTX images were co-registered to the HRSC ORI using ACRO v2.2. The yellow-coloured image in the top-left part of the figure is CTX image D14_032511_0959_XI_84S078W_ORI (Ls 345) while the blue-coloured image in the centre is the CTX image B11_013813_0955_XN_84S078W (Ls 299). The background image is the level-4 HRSC ORI H2288_0000 (Ls 312). All images are shown using a polar stereographic projection.
WP5 iMars web-GIS Software
iMars created a repository of raster data from ESA and NASA missions covering the last four decades of Mars orbital exploration. With these data, systematic surface change surveys can be performed using auto co-registration and data mining techniques. This will lead to a better understanding of surface processes and relative ages of different areas.
Each dataset is inherently associated with its imaging acquisition time. Thus, a stack of co-registered images will allow navigation through time layers using time sliders and associated techniques. In order to make these data accessible to public users as well as researchers, a webGIS platform was created by FUB, which offers interactive navigation and dedicated visualisation facilities. It builds upon a spatial database management system and an open-source Web Map Service provided by an open-source webserver backend.
Within WP5, FUB has developed a planetary web-GIS able to handle massive amounts of data and various different types of data layers. We have achieved our goals for an intuitive interface with a high performance display of the different vector and raster data types. An overview of the overall client-server based system design is shown in Figure 12 [System design of the iMars webGIS consisting of the frontend (including a regular web server) and the backend with its several storage subsystems and processing components.].
The system consists of a frontend part, which is essentially the web page, and JavaScript files, which are loaded by the user’s browser. They are directly located on a webserver. The backend parts get loaded indirectly from the browser and consist of a single or possibly multi-server setup of a web server with an attached tile cache and a web server with attached raw data storage. The database system can be located on a separate system for load balancing purposes. The load balancing design enables performance scaling for high user demands.
The user interface as part of the frontend in the browser consists of several modules built upon the open-source OpenLayers library. We have developed additional control panels to interact with the map and its contents. Figure 13 [Overview of the existing and newly developed graphical user interface elements] gives an overview of the available components. In particular, the newly developed components are:
1. Modified Layerswitcher module (derived from the existing ol3-layerswitcher project on Github). New functionalities such as group folding/unfolding, group selection, transparency sliders, information buttons, layer sorting and manual re-ordering and layer deletion have been added.
2. Toolbar for the selection of single images from the respective footprint coverages and for the creation of dynamic layers for each of the single co-registered image. A play, pause and stop button group is available for iterating over a time series of images.
3. Time panel for the filtering of the available coverage data for time. Both universal time and Martian time fields are available for the start and stop time entry. Additionally, the time span can be selected with a time slider.
4. Projection switcher, a small element to choose between the three different projections for global equidistant cylindrical (equirectangular), south polar stereographic and north polar stereographic.
5. Goto lat/lon is a two-field entry mask for entering individual latitude/longitude coordinates – the map centre moves to the entered values upon execution.
The remaining controls in Figure 13 are regular Openlayers controls for full-screen toggling and zooming functionality.
The work in WP5 consisted also in the assembly of various data layers for visualisation in the webGIS and in GIS services. In particular, the webGIS offers the following data layers (bottom to top in the webGIS):
1. A combined topographic group layer consisting of several layers of DTMs of different instruments (HRSC, CTX, HiRISE) as the result of work packages WP1, WP3 and WP4. The DTMs are displayed as colour-coded hill-shaded maps. The single topographic layers with their different cell sizes integrate seamlessly into each other so that the user can zoom in and out without recognising the transition between different datasets. Despite that the single datasets can be turned on and off as single layers to be able to investigate single instrument’s data products exclusively.
2. A group layer of the HRSC image mosaic products as released by the HRSC team. The group contains the grayscale mosaic of MC11 in 12.5 metres per pixel as well as the colour mosaic in 50m per pixel. Both layers are equipped with transparency sliders to enable the user to view the colour-coded topography information together with the images. This data is currently exclusively visible on the iMars webGIS server.
3. Co-registered ortho-rectified images of CTX and HiRISE from different providers within WP4.
4. Vector layers for generic informational needs, such as a quadrangle overlay to visualise the borders of the quadrangles, a landing sites layer with all the past successful and planned landers on Mars, and a nomenclature layer with the official feature names of the IAU working group for planetary system nomenclature (WGPSN).
5. Footprint layers of the available NASA and ESA image data on Mars connected to the geometric database as described above. The footprint layers allow spatial queries and attribute information via on-screen pop-ups. Within the attributes are links to existing sources of the original datasets for further information and download. The footprints can be filtered using a slider-based selection tool to set start- and end-points in time.
6. Additional footprint layers are available for the existing automatically co-registered single image data created in WP4. These footprint layers serve as a selection database for visualising the available time series of overlapping single images for visualisation of change detection.
7. A colour-coded high-resolution coverage map produced within WP6 representing the available repeat imagery from all available NASA and ESA image instruments for the visualisation of sweet spots for change detection observation.
Figure 14 shows a typical session within the webGIS with selected footprints filtered by a specific time range above the HRSC multi-orbit mosaic from MC11. With the Info button enabled, the user has clicked on a footprint to query attribute information.
Besides this browse and query functionality, the iMars webGIS offers an additional workflow for the visualisation of change detection. It is applied to the data from WP4, the automatically co-registered orthorectified image data, and the HRSC single images. The unique aspect of the tool is the loading of dynamic layers of single images. To be able to control the single image granules from within the frontend with JavaScript, every image has to be loaded as a single layer from the server. The single image loading workflow is controlled via the Toolbar. After enabling layers from within the “orthoimage footprints” group, overlapping geometries can be selected using the “single select” tool. After the selection, the dynamic layers can be added by the “create layers from selection” button. A new group layer called “Orthorectified images” will appear on top of the layer list populated with individual layers for the selected single images. By hovering over the list of layers, the image time will be shown. The user can re-arrange the list manually by drag-and-drop and automatically according to image name, acquisition time or image resolution. The sequence of images can then be played in an alternating cycling loop using the “Play” button (see Figure 15 Snapshot of the visualisation of time series of co-registered single images from the MOC, HRSC, THEMIS and CTX instruments. When the play button is active, the visible image is altered every half second from the list of available images in the group layer). The shown technique proves to be a very good way of visualising surface changes as the difference of features between the alternating images is immediately recognised by the observer.
The large number of images as well as their individual sizes poses a severe challenge to such a system. We have included a dedicated caching system in the pipeline to effectively save image tiles of all types of layers. By the use of image tiles instead of complete images loaded from the WMS, the client will also save the tiles internally which makes the user experience more fluent during panning and zooming. On the server side, we implemented an adapted caching instance to collect and keep the tiles on the server.
After completion of the main targets for the web-GIS functionalities and the ingestion of the products delivered by DLR, UCL and UoS, a performance test of the FUB housed development system was carried out. Performance collection software was used to measure and visualise the simultaneously placed system load of the server machine. It was found that even on the limited developer hardware, the system was usable with ten concurrent users and highly demanding tasks. On the other hand, the application appears as an expert tool with many sophisticated functions which demand high performance. The target group of planetary science expert users is expected to take their time to view the “show-and-tell” YouTube videos and read help pages before or during the use of the software. It should be mentioned in the documentation that the user has to expect long delay times for certain functions. For example, the time of loading many dynamic images may last tens of seconds even on very fast hardware, but the users are “rewarded” with the time-based flickering function afterwards. As long as the end users are well informed with proper documentation, we can expect that they will anticipate and accept delay times whilst working with the application.
An enhanced prototype of the web-GIS was implemented and finalised by the end of 03/2017. A developer environment version of the system with password-protected access was set up (http://imars.planet.fu-berlin.de/dev). Additionally, a public version with password-free access to the internet has been integrated into the consortium’s web page (https://arquivo.pt/wayback/20170603214104/http://www.i-mars.eu/web-gis/) hosted at UCL-MSSL. All the functionality has been tested and proven to perform in a stable manner in a simultaneous multi-user environment.
All the data provided by the project partners were fully incorporated as GIS layers. In addition, the FUB web-GIS also served as an internal data evaluation tool for the main parts of the project with regards to stereo coverage, radiometric image quality, co-registration offsets, processing artefacts and topographic height offsets.
Finally, a QGIS plugin for interactive selection and display of MARSIS and SHARAD radargrams was developed in Python by EPFL and is published in open source (via the GitHub). This operates on multiple OS platforms such as Windows, Linux and MacOS.
WP6 Change detection from Data mining & validation
WP6 is focused on the automatic detection of surface changes on Mars. Change detection processing was conducted using three independent pipelines, developed by UCL, DRL and UoS, respectively. The input data, i.e. high-resolution co-registered images and DTMs processed using the techniques developed in WP1 & 2 and processed in WP4, were processed offline and independently by each partner. Moreover, part of the results of the change detection processing was released by UCL to UNOTT so as to populate their change detection pairs for use within the WP7 crowdsourcing experiment.
A generic automatic change detection approach, developed by UCL, focuses on a batch-mode automatic detection of surface changes on Mars. This pipeline was designed to be context-free, i.e. to detect semantically meaningful changes regardless of the specific type of change. The inputs for this processing pipeline are images, co-registered and orthorectified to HRSC baseline using the UCL ACRO pipeline. As a result, the resolution limitations of the aforementioned HRSC imagery, excluded HiRISE imagery from the co-registered datasets, thus limiting the change detection input to imagery acquired by CTX, HRSC, THEMIS-VIS and MOC-NA instruments with IFOV≥1.5m.
The techniques developed have demonstrated the potential of planetary surface automatic change detection pipelines, thus generating a new family of tools that can assist the understanding of planetary surface dynamic processes. Apart from their clear technological contribution, these techniques have already assisted in the discovery of unreported instances of Mars surface changes, which can be used to augment the catalogues of current day Mars surface activity. In many cases these can be employed to improve numerical GCM models of the Martian atmosphere.
The input of the UCL generic change detection algorithm are pairs of overlapping images, that have been co-registered using the automatic co-registration and orthorectification (ACRO) method that has been developed within the iMars project. For each region, a set of overlapping pairs are selected and pruned to discard image pairs with overlapping regions that are smaller than 256 x 256 pixels.
Subsequently, the preliminary set of input pairs is further processed to extract a temporal series of images, thus eliminating pairs of images that are not temporally adjacent and processing only the minimum number of image pairs. This approach is required to optimise the computational resources without missing any change instances, since by default any change can be temporally narrowed down to a pair of temporally adjacent images (since, if a change can be found between two images I1 and I3 and there is another image I2 that was taken after I1 and before I3 then the change can be found either between I1 and I2 or between I2 and I3). Temporal ordering is particularly useful in areas that have been mapped multiple times, since it reduces the computational complexity from quadratic O(N2) to linear O(N), where N is the total number of images mapping the area. A schematic diagram of the UCL change detection pipeline is shown in Figure 16.
Every remaining pair is fed into the change detection pipeline to automatically identify context-free semantically meaningful changes. Each corresponding HRSC DTM (used during the ACRO pipeline for orthorectification) is also fed in the input, so as to assist in the detection of non-informative changes that are caused by differences in the illumination conditions resulting in changes in the appearance of the surface. Before this step, the input images are initially compared so as to estimate pixels with significantly different pixel values. Because automatic change detection software can estimate large-size changes or medium-size changes, but not changes that take place in a very limited area, changes that happen within “isolated” pixels are discarded and only change pixels that cover a large area are further taken into account.
The next step of the algorithm is to discard pixel differences that are caused by different illumination (e.g. shadows). The corresponding patches are ignored from further processing, while the rest of the (candidate) patches are fed into a deep learning module that can detect four different types of changes. Note that the detectable “types” of changes are defined according to their low-level image content, which only loosely correspond to semantically defined types of changes. The four change detection modules are as follows:
• Blob detection, changes like homogenous blobs are present in only one patch of the pair
• Texture detection, where changes that appear like small-sized changes in the patch texture
• Global detection, which detect global changes between the two patches
• Motion detection, in which a small feature has moved between the two patches
Each module is a supervised learning classifier that produces a “confidence level” score, i.e. a value between 0 and 1 that models the probability of a change of a certain type is present. These intermediate scores are further combined (using a second-layer classifier). The final result for each patch is a score, which is compared with a threshold to declare a change. Finally, the detected changes are expanded so as to cover a 512 x 512 region, so as to provide context for the identified dynamic feature.
The UCL change detection algorithm has been extensively used to detect changes on the imagery that was co-registered and orthorectified within the project, as part of WP4. The change detection results that were acquired using the algorithm of the previous sub-section are as follows:
• 3,365 changes on image pairs from randomly selected image pairs from MC11-E, MC11-W, as well as regions-of-interest all over Mars, a dataset that was also internally released so as to form the input to the WP7 crowdsourcing experiment
• 465 changes from the same datasets that were further manually annotated, generating 270 true positive and 195 false positive results. The implied change detection performance (58.06%) provides a first evaluation of the change detection performance, while a more thorough evaluation will be conducted with the help of planetary scientists and the citizen science inputs. Note that while this score doesn’t appear impressive it still defines the state-of-the-art, since it is the first time that such an algorithm has ever been applied to Mars. Moreover, from discussions with planetary science colleagues, an automatic tool that generates 3 semantically meaningful changes per 5 detected change instances is useful.
• 3,512 changes that were detected by the available high-resolution imagery of the MC11-E half-quadrangle. Note that 527 of them were duplicates with the first set of experiments, therefore they needed to be excluded from the overall number of changes. However, the fact that the algorithm produces duplicates signifies that the results are reproducible, i.e. the stochastic stages of the algorithm don’t cause irreproducibility in the algorithm output, a feature that needs to be carefully designed in any algorithm that employs stochastic stages.
• 4,764 changes that were detected from the available high-resolution imagery of the MC11-W half-quadrangle, 1,089 of which were duplicates with the first set of experiments and needed be excluded from the total change estimation.
Overall, the change detection runs estimated 10,490 in the MC11 area that is 5.5% of the Martian overall surface. Taking into account the preliminary true positive rate of 58.06% and extrapolating to the total size of Mars, this would mean more than 110 thousand instances of change, could be detected in all the imagery acquired by CTX, HRSC, THEMIS-VIS and MOC-NA instruments.
This number confirms the main hypothesis of the project, i.e. that the frequency and distribution of Mars surface changes is so large that any “manual” analysis of image pairs to track dynamic features is unrealistic due to the data-intensive Mars exploration of the last two decades, and that Mars surface science would greatly benefit from the development of automatic change detection techniques that would assist planetary scientists to flag the most “promising” dynamic feature regions. Moreover, it becomes apparent that a thorough examination of 110,000 changes would require a significant amount of time from the planetary science community. Some examples of automatically detected changes are shown in Figure 17 [A new impact crater that was automatically detected in MC11-E half-quadrangle. Images (a) and (c) are THEMIS-VIS while images (b) and (d) are CTX images. The automatic change detection pipeline detected the pairwise change in all pairs (i.e. (a)-(b), (b)-(c) and (c)-(d)), thus successfully tracking the whole process of the impact crater formation. Image (a), which is the last image before the impact, is from 2005 while image (b), which is the first image after the impact, is from 2007. This is a new discovery, not reported in the literature, which becomes more important when considering that the impact happened only 190 km from Opportunity rover. See also Figures 18 a & b [New slope streaks detected over Olympus Mons Aureole].
The change detection pipeline development was shown to be a rather challenging task on its own, given the fact that it required first the successful completion of WP2, WP4 and WP7. The final results of the analysis are going to be initially manually checked at UCL to generate a thorough analysis of the algorithm.
Subsequently, the results will be sent for further assessment to planetary scientists, who will provide a valuable feedback about the results and pinpoint unreported instances of Martian dynamic features, and possibly newly discovered dynamic features. We believe that it is straightforward that the impact that such a dataset can have on the Mars scientific community. This will be one of the main legacies of the iMars project, i.e. that multiple future publications about the Mars surface dynamic processes will originate from the results of the data-mining algorithm that was developed within this project.
In WP6, UOS efforts were concentrated on addressing arguments regarding the migration speeds of Martian dune fields and their correlation with atmospheric circulation. In this aspect, UoS developed a generic procedure [see Figure 19: UoS Implemented algorithms for the observation of dune migration] to precisely measure the migration of dune fields with HIRISE employing a high-accuracy photogrammetric processor and subpixel image correlator. The key features over conventional manual tracking which has been employed by planetary scientists are:
• The introduction of very high resolution orthoimages and stereo analysis based on hierarchical geodetic control for better initial point settings
• Positional error removal throughout sensor model refinement with a non-rigorous bundle block adjustment, which makes possible the co-alignment of all images within a time series
• Improved sub-pixel co-registration algorithms using optical flow with a refinement stage conducted on a pyramidal grid processor and a blunder classifier.
The algorithms were tested on time sequences of high-resolution HIRISE images over a large number of Martian dune fields (see example in Figure 20: Example of the detected migration pattern over the Kaiser dune).
As one of the pilot studies on change detection agreed within the consortium, DLR analysed multi-temporal HiRISE images from the edges of the north polar cap of Mars in order to detect and study changes due to block falls and their implications for processes involving active mass wasting. After a systematic survey of recent block fall events recorded in HiRISE images, this study included an assessment of the geometric quality of the image data products used and the resolution limit of the blocks in images and DTMs. In a next step, detected events were grouped according to their imaging times and conditions, and methods for automatic detection of blocks were tested and applied, with the aim of better defining the sources of the unstable blocks and derive estimates on erosion rates.
WP7 Crowd-sourced features for change discovery and validation of data mining
WP7 concerned the development of Citizen Science activities for using the imagery developed by UCL for WP2, WP3 and WP4 to investigate changes on the Martian surface across the time period during which orbiting spacecraft observations have been undertaken. The intent of WP7 was to provide a means to engage interested members of the general public with these data in a structured manner (relevant to dissemination and iMars WP8 and also to complement the web GIS system developed in WP5), to provide scientific results to support on-going development of algorithmic change detection techniques (WP6) and to extend knowledge about the design of citizen science projects including the provision of guidelines and processes for developing future projects. WP7 introduces a notable interdisciplinary element into the project.
The specific scientific aims were to bring Human Factors knowledge to bear on the novel problem of Martian citizen science, to carry out experiments with humans in support of design decisions about citizen science tasks, to build a citizen science project and develop relevant software as required, validate results using it and to finally report guidelines and results based upon this.
At the start of WP7 UNOTT undertook an extensive review of the literature related to citizen science. This remains a relatively unstudied area in its own right (with some noteworthy exceptions) although significant relevant knowledge exists across a wider multidisciplinary terrain including Human Factors, Computer Science, Cognitive Psychology, Psychophysics, Organisational Theory and Digital Economy. The core reason for this emerges from the recognition that citizen science is the development of an online system of work (albeit work undertaken by volunteers). It involves the optimisation of motivation and performance, the deployment of tasks, designing interfaces that consider human psychological capability and relating all these to the primary scientific goals of the work. In order to bring this diverse literature together, to structure our own design process and to exemplify this novel understanding of the problem space, we devised the model shown below that shows how different facets of citizen science design interrelate and constrain each other and questions of motivation and performance trade-offs should be considered (see Figure 21: A Human Factors system model of Citizen Science).
This model of the functioning of a citizen science system implies a particular form of a development process that emphasises user-centric processes and the representation or advocacy of that user at all points in the process, thus ensuring a balance between the technical and social elements of the resultant sociotechnical system portrayed above. The development process takes a deliberately phased process to ensure that, as the complexity of the system grows, key human aspects remain in focus (see Figure 22: A nine stage citizen science development process used within iMars. Examples of key considerations and tradeoffs are listed alongside each stage with examples from the present work. Note at the third stage that the presence of automation should emphasise rather than detract from the consideration of human issues which become all the more crucial). Again, to our knowledge no standard process or commensurate set of detailed guidance has been identified hitherto in this area. Further discussion of this work, together with our reflections and is available in iMars public deliverable D7.4 and from the specific viewpoint of Human Factors practice in Houghton et al., (2016).
As per the development process below, a specific design issue in this project is how to best design this for change detection. However, change detection has to our knowledge not been a particularly popular form of citizen science task (which tends more often towards annotation, transcription or some form of counting) with little direct precedent. Therefore we carried out a series of experiments to explore key parameters in this activity; how many features a participant should be asked to look for (should they specialise or consider a wide range?); what interface should be used (side-by-side vs. flicker, and how should that flicker be tuned?); should participants know anything about crowd or algorithmic confidence before viewing stimuli or go them afresh; and how comparable is the performance of amateurs and professionals in these tasks? Alongside these quantitative measures we also collected data about perceived workload, trust in the system and qualitative feedback formative to the next iteration. The findings of these experiments are summarised in D7.4.
The final form of the citizen science project was developed iteratively and required in its earlier stages the development of specific functionality to display ‘flickered’ imagery and handle the translation of iMars imagery into a form that could be used within the Zooniverse Panoptes framework. This is accessible in public Github form at: https://github.com/sprinklebum/iMars-eu
The iMars citizen science project (Mars in Motion) can be accessed at: https://www.zooniverse.org/projects/imarsnottingham/mars-in-motion
Key functionality offered by the site, as a result of the development process we followed includes: the ability to manually flicker, automatically flicker and where required by users in the interests of wide accessiblity, compare side-by-side iMars image pairs [see Figure 23: Interface for change detection]:
Social functionality includes the provision for users to comment on features, collect particularly noteworthy or photogenic imagery and to discuss with other users their experiences of participating in the project [see Figure 24: The collection of interesting imagery as a facilitator of participant interaction].
Data from the first 9,000 classifications using Mars In Motion are shown in Table 2, broken down by feature type.
Feature Type N.Detected Mean Volunt. Agr-t Median Volunt. Agr-t Average N.of Volunt.
Dune 105 0.77 0.77 2.07
Dust Devil Track 170 0.84 0.84 2.88
Gully Slide 65 0.53 0.52 2.77
Impact Crater 83 0.63 0.63 2.27
Seasonal Fan 32 0.50 0.50 2.00
Slope Streak / RSL 154 0.75 0.75 2.59
Table 2: Classifications of the first 9,000 change detection images broken down by feature type.
Some examples of changes that have attracted the most interest by volunteers to-date are shown in Figure 25 [Examples of volunteer change detections. From top right clockwise: (a) dust devil tracks, (b) slope streaks, (c) & (d) new impact craters], which include new impacts and dust devil tracks.
WP8 Outreach
WP8 aims at maximising the visibility of the project outcomes with the scientific community and the general public as well as identifying new opportunities which arose as a result of the datasets and the web-GIS and crowd-sourcing mechanisms. WP8 includes three user workshops and the documentation of their implementation and results (tasks T8.1-T8.3). Task T8.4 focuses on the production of animations and videos to present the developments and results of iMars in support of their dissemination. Finally, task T8.5 comprises the set-up and maintenance of the project website by UCL.
A total number of 10 abstracts were submitted to the DPS/EPSC 2016 of which 5 were accepted as oral presentations at the meeting and the remainder as poster presentations. Furthermore, the consortium tried to arrange with the meeting and science organizers to bundle the presentations in one session to highlight their common links to the overall approach of iMars. Unfortunately, this was not granted by the program committee as contributions are accepted and arranged according to individual sessions at DPS conferences with no flexibility to have any special sessions.
Nevertheless, the iMars contributions covered the entire range of topics dealt within the project. Spreading the presentations over different sessions, on the other hand, provided also an advantage, as a wider audience was reached.
The Co-ordinator attended some of the oral presentations. The talk given by Jessica Wardlaw (Sprinks et al., 2016a) was well attended with some 30 people, the talks from Lida Fanara, Jung-Rack Kim and Yu Tao had almost a 100 attendees with lots of follow-up questions raised. The poster session was poorly attended with only 2-3 visitors/poster. Physically it was away from the main part of the poster room and away from the refreshments. Nevertheless, it was a good opportunity to engage with NASA scientists about the iMars results.
The RPIF 3D workshop (T3, WP8) was organized by MSSL/UCL in co-operation with the RPIF UK facility (Director: Prof. Jan-Peter Muller & Data Manager: Dr Petet Grindrod) and Europlanet (Prof. Steve Miller & Dr. Norbert Krupp). Its main objective was to introduce early-career scientists to the range of software tools available to generate 3D data products from planetary exploration data, mainly using data of Mars. In addition to known software libraries and workflows like the NASA-USGS ISIS and Socetset® solution, newly developed software like the modified UCL-NASA Ames Stereo Pipeline, which was developed within the iMars project, was demonstrated. Other topics of this 3 full day workshop included data handling and data fusion, tools for the digitisation of geological and geomorphological features and an introduction to GIS systems. In total 22 international students and young professionals participated in the workshop including 1 participant from Turkey, 4 from Russia, 1 from Spain, 3 from Poland, 1 from Hungary, 1 from Slovenia, 1 from Italy, and 10 from the UK. All these participants volunteered to participate in the beta testing of the Mars in Motion system. All aspects of the iMars project were covered in the presentations, which included:
• Introduction of new datasets from the HRSC experiment
• Co-registration of different NASA data sets to HRSC
• Processing of stereo data into DTMs and data mining
• GIS presentations and tools
• Introduction of Citizen Science platforms for scientific research.
Next to the demonstration of the iMars products, a questionnaire was developed to be answered by the workshop participants. Answers provided guidance on further steps within the project and its dissemination as well as potential spin off projects.
22 participants of the workshop filled in the questionnaire that was handed out at the beginning of this meeting. Expertise and fields of research of the participants was wide spread with a concentration on planetology, geology and GIS [see Figure 26: Distribution of participants’ interests & study fields (left). Right – data participants worked on before the workshop]. All, except two of the participants do work with planetary image data of various sources. However, the majority did have contact with Mars exploration data of some kind. Here it was indicated that the highest resolution orbital imaging data (CTX and HiRISE) are the most used data, followed (as far as Mars data are concerned) by HRSC images from the European Mars Express mission and rover images of NASA’s MSL. The category “Others” includes Martian spectral data as well as orbital data of missions to other planetary bodies like Messenger data or Cassini data.
Based on the questionnaire answers, for most participants of the workshop the focus of their studies appears to be the interpretation of data and data products rather than producing derived products like DTMs themselves [see Figure 27: Fields of application indicated by the workshop participants]. Though 16 out of the 22 participants need to process retrieved data to achieve their tasks.
Data processing in this case includes co-registration, preparatory work to load the data into software, i.e. ArcGIS, radiometric calibration, and also mosaicing to extend the covered area to the area of interest. All this is done for the purposes of final DTM generation, mapping, mosaicing and detailed interpretation of the data. Visual inspection of input data to the processing appears to be a common method to judge the suitability of data for the purpose. Though, some participants do use the meta-information to select their input data.
Finally, 11 short videos were created to promote activities and results of the iMars project beyond the end of the project. These comprised descriptive videos, tutorials, and virtual flights over the reconstructed Martian surface. The videos demonstrated results of the iMars project, and described the developed tools and techniques used to derive the data products. These 11 videos were placed on a dedicated YouTube Channel that can be accessed through https://www.youtube.com/channel/UCnQ_SviuCnJjSQOG8v-Ru1Q as well as through https://arquivo.pt/wayback/20170603225639/http://www.i-mars.eu/outreach/media/.
Potential Impact:
Impact Summary
The data product specifications for HRSC multi-orbit products will form the basis for the systematic processing of multi-orbit HRSC data products by the Mars Express HRSC team. The quadrangle mapping by HRSC multi-orbit data products represents a major effort in the cartography of Mars, performed by the Mars Express HRSC Team using the complete mission data record, and will be available to the science community and general public through the ESA Planetary Science Archive (PSA) and NASA Planetary Data System (PDS). The results of the investigations performed within iMars on quality-related aspects and validation have been directly communicated to the HRSC Team.
UoS efforts and subsequent research work based on the outcomes and softwares of the WPs contributed to the planetary science community included (a) demonstrating the importance of precisely controlled topographic products over a planetary surface for quantitative analyses as well as for simulation, measurement and modelling; (b) how to validate the conventional tools in planetary surface interpretation with other approaches. For instance, UoS proposed the validation of previous results of martian dune migrations based on a new method. It identified many highly important missing points, which was caused by the absence of the technological understanding by the researchers.
Technical foregrounds from UoS, especially WP1 and WP6 are already being applied to addressing certain terrestrial environmental issues. The software routines developed by UoS efforts for WP1 stereo processor implementation have been applied for a Greenland mapping project. A number of spaceborne optical stereo and radargrammetric DTMs over southern Greenland were created using a variant software based on WP1 UoS stereo line and used for the scientific research works. WP6 software by UoS was tested over the Kubuchi desert in northwestern China where dust storms frequently occur and cause significant problems in public health over all of northeastern Asia. The initial test was very successful and the further scientific utilization of WP6 UoS software has gained the support of an international organization.
For the real world applications of foreground by UoS, especially HD video contents and their interfaces, UoS has contacted potential industrial partners and public bodies as the possible consumers. Although it still requires more investments to achieve realistic outcomes, such efforts clearly arouse more public/industrial attention for the international Martian studies and their applications.
The CASP-GO, ACRO, and change detection software developed in WPs 1 & 4 and WPs 2 & 6 are a valuable new resource to the scientific community of geologists who are interested in the exploitation of DTMs and co-registered image sequences showing changes over the last 40 years. They have huge potential for EO imagery as well as application to other planetary bodies such as Mars and the Moon. CASP-GO will be released on GitHub for maintenance by NASA Ames later in 2017. The NASA data products from WP4 will be converted into PDS4-complaint format and released through the NASA PDS Imaging node at JPL whilst it is planned that the new data products from HRSC for the SPRC will be released through the ESA PSA. The webGIS will be the principal means whereby members of the scientific community can visualise and access iMars derived products until they have been integrated into the PDS and PSA systems. Some of the iMars products are still in production as they do not require very much manual supervision or intervention, such as completing the DTM production of all the CTX stereo-pairs with 100% overlap, all the HiRISE stereo-pairs which have ≥5 repeat images over the same location and if some mechanism can be found for student interns completing the production of the overlapping NASA images over some 45% of the planet which HRSC has mapped to date. Change detection is likely to require new resources in the future to be applied to the remaining 45% of the planetary surface. Initially, it is expected that most science exploitation will take place through personal collaborations but it is anticipated that over time as the community learns of the value of the resource that greater and greater use will take place independent of the iMars consortium, especially once the data are integrated into the PDS & PSA infrastructure.
The findings of the work into the design of citizen science have wider application in the area of ‘human computation’. In contrast to citizen science, human computation describes the paid execution of small decomposed ‘task units’ by humans using online platforms such as the Amazon Mechanical Turk and Crowdflower. This might include activities such as annotating documents, looking up information on the web, translating short phrases and detecting features in imagery (similar tasks are obviously undertaken under the rubric of citizen science albeit for different reasons). The design of these platforms is a topic of some controversy and debate, particularly owing to concerns about the economic fairness of payment schemes and the experience of workers themselves, particularly as the extremes of task decomposition employed, leave workers doing fragmentary, momentary and generally unrewarding work without much autonomy, community or a sense of the wider impact of their labour. As Kittur et al., (2013) ask: “Can we foresee a future crowd workplace in which we would want our children to participate?” The root cause of these problems is that the work is not as such designed for humans, as designed around what cannot be otherwise automated (this is known as ‘left-over task allocation’). This is arguably not dissimilar from a digital revisiting of Taylorism and Fordism, historically forms of work design associated with worker demotivation, drudgery, strained industrial relations and flawed attempts to reconcile human activity with that of a factory machine (see Sharples & Houghton, 2017 for a discussion). We are reminded of the latter forcibly by accounts of online work that refer to people in dehumanising terms such as ‘human processing units’, conceptualising them not as machinery now but as elements of computer architecture. Recast from this perspective, work in the related area of volunteer citizen science allowed us to examine some of these issues in online task execution and its relationship with automation in a far more benign context where the onus was on us to please volunteers rather than for workers to please employers. If, as many predict, human computation expands as a form of work as a concomitant to increased automation in the workplace, the humanistic and user-centred ways of thinking about online work and the development processes developed and demonstrated within this project will be useful correctives emphasising the role of the human and providing a design process towards a human-automation collaboration that delivers good task performance for the mutual benefit of workers, businesses and citizens.
Dissemination activities carried out during the project were coordinated in a dedicated WP, which included contributions from all project partners. Dissemination activities were aiming at increasing the visibility of project aims and results to the science community and general public, at identifying new opportunities which arose as a result of the datasets and the web-GIS and crowd-sourcing mechanisms, and at implementing efficient communication with the scientific community and general public for enhancing and promoting the use and benefit of iMars tools and data. An additional aim was to employ the NASA Regional Planetary Imaging Facilities (RPIFs) to promote the products and tools developed by iMars. This required continuing efforts over the entire lifetime of the project and was achieved by implementing (a) a series of user workshops and a dedicated science session at a major planetary science conference (EPSC 2015) (b) a series of videos accessible through a public on-line channel (c) the iMars website https://arquivo.pt/wayback/20170603204309/http://www.i-mars.eu/
The prospects for future application of iMars topics and tools show high potential according to feedback gained within the project through several dissemination events and exchange of experience ideas with potential users through presentations, questionnaires, and workshops. Since the developments accomplished in the project have been flanked by several dedicated user workshops, including evaluation of user feedback returned, an enhanced compliance with user requirements and user interests can be expected. The high interest in the community for the topic of surface changes was confirmed throughout the numerous presentations of the project at workshop and technical sessions of planetary science conferences. The plan for a wide and unrestricted access to data products through the PSA and PDS archives will further support future studies in the area. The project has also achieved a high impact on public awareness for current space science and its methodology through the realisation of a citizen science study and its promotion via the different dissemination events.
Three user workshops where held during the project:
• User Requirements Workshop, at the European Geophysical Union (EGU) General Assembly, May 2014, Vienna (Austria)
• First User Consultation workshop, at the European Planetary Science Conference (EPSC), September 2015, Nantes (France)
• Final User Consultation workshop, at UCL (Great Britain), June 2016, in conjunction with the RPIF 3D workshop
The objective of the first user workshop (User Requirements Workshop) was to introduce the project’s aims and developments to the scientific community and in particular potential users, and assess user requirements and interests. The specific objective of the second workshop (First User Consultation workshop) consisted in presenting already accomplished project developments and obtaining user feed-back to guide the further project developments. Finally, the objective of the third workshop consisted in presenting the project results to users, gain feed-back to guide final implementation and analysis activities as well as evaluate exploitation opportunities. The workshop included presentations on the contents and aims of iMars, initiation of a user survey developed by he consortium, and open discussion with potential users recorded by taking minutes. The survey was accessible in a printable version as well as an on-line version hosted on the iMars website. The workshop minutes and the survey responses were analysed and documented in a report, which was made publicly accessible on the iMars website. The conclusions concerning user requirements were considered as useful guidelines for the technical development tasks of iMars. Experience was gained concerning the expected return of user surveys involving technical aspects of particular relevance, in terms of number of responses. The high interest in the community for the topic of surface changes was confirmed and the workshop and survey helped to define a set of specific case studies in this field and obtain feedback form researchers actively engaged in the area.
The second workshop was organized in a special technical conference session and a demonstration event at EPSC 2015. It was organized jointly with FP7 projects CROSS-DRIVE and PRoVIDE to highlight the complementarity of approaches and maximize the outreach to a potentially large user group. The direct feedback gained from discussions during the technical session and the demonstration session were documented in a publicly accessible workshop report together with an evaluation of “lessons learned” and updated information on user requirements. The technical session was part of the official conference technical program (MT8, Zooming in-and-out of Mars: new tools to interact with multi-resolution Mars datasets, conveners K. Willner, J.-P. Muller, S. Gupta) and included 11 oral presentations and 8 posters in two scientific sessions on Sept. 29 and 30, 2015.
The third workshop consisted of a hands-on workshop organised in collaboration with the UK NASA RPIF at Mullard Space Science Laboratory (MSSL) in June 2016. It was held in conjunction with the EUROPLANETS 2020 NA1 networking activity hosted at UCL-MSSL and led by the Director of the UK NASA RPIF, Prof. Muller on 3D Imaging. All of the 2nd day and part of the 3rd day (8-9 June 2016) were devoted to showing-off the iMars system, including CASP-GO, ACRO, the webGIS, data mining and getting users feedback to a questionnaire developed by the consortium. The 3rd day of the workshop included capturing the scene classifier data required for WP6 using the prototype “Mars in motion” website. The audience of the 3rd workshop consisted mainly of early-career scientists from different European and non-European countries (1 from Turkey, 4 from Russia, 5 from Inclusiveness countries, 1 from Spain, 10 UK).
The final outreach event organised in London, took place on March 14, 2017, 14 presentations on iMars tools and applications, presented by iMars partners and external colleagues to an audience of 70 interested members of the public and colleagues from the planetary science community.
A specific effort within WP8 was devoted to the production of a collection of 11 videos accessible on a public on-line channel. The topics addressed by the videos include the description of tools developed within iMars, the explanation of techniques applied, and the presentation of data products. Examples include the demonstration of the new HRSC colour products in 3D for 3 different areas, a video showing how mosaics are produced and a flight over the new MC-11 mosaics, a video demonstrating 3D interactive visualisation tools being used to show off the new co-registered HRSC, CTX and HiRISE 3D products and the EU-FP7 PRoViDE interactive web-based tool, Pro3D® developed for 3D digitisation of HiRISE 3D products. In addition, videos to show the CASP-GO, ACRO and data mining tools were developed as well as for the QGIS extensions to allow the visualisation of SHARAD and MARSIS subsurface radar echograms. Finally videos were produced to show the citizen science “Mars in Motion” tools.
List of Websites:
https://arquivo.pt/wayback/20170603204309/http://www.i-mars.eu/
iMars Co-ordinator: Professor Jan-Peter Muller, j.muller@ucl.ac.uk
Executive Summary
The iMars project focused on developing tools and value-added datasets to massively increase the exploitation of space-based data from NASA and ESA Mars mission imaging and 3D data beyond the PI teams. iMars has added significant value by creating more complete and fused 3D models of the surface from multi-resolution co-registered stereo all co-registered to a global reference system derived from laser altimetry and has shown how these 3D models can be employed to create a set of co-registered imaging data through time, permitting a much more comprehensive interpretation of the Martian surface to be made.
Emphasis was placed on the co-registration of multiple datasets from different space agencies and orbiting platforms around Mars and their synergistic use to discover what surface changes have occurred since NASA's Viking Orbiter spacecraft first went into orbit around Mars forty years ago.
iMars brought together the best expertise in Europe for the processing of Martian orbital data within a single environment for handling, visualising and interpreting these data. The ESA Mars Express High Resolution Camera (HRSC) provided the 3D mapping products used as base data (for around 50% of the surface), where possible. When CTX stereo products are also available over the same areas as HRSC (for around 20% of the surface), then the CTX products can be co-registered with HRSC and CTX 3D mapping products can be employed as the base data for higher resolution images such as MOC-NA and HiRISE. A Co-registered Ames Stereo Pipeline using Gotcha Optimisation (CASP-GO) was developed for large-scale production of CTX 3D mapping products and small area production of HiRISE products. Some 5,300 CTX stereo products have been processed using cloud computing provided by Microsoft Azure® covering about 20% of the Martian surface. In iMars, standards were set for the production and dissemination of HRSC mosaiced products, which are easier to utilise for co-registration than individual strips and have better internal geometry. A fully automated Auto-Coregistration and Orthorectification (ACRO) system was developed to operate on a linux cluster without any manual intervention. Around 15,000 NASA images (out of the ≈400,000 acquired with resolution of ≤100m) were processed using the ACRO system covering around 4% of the surface. New HRSC 3D mapping products were produced for the South Polar Residual Cap area and the unfunded (by iMars) PhD student is working on applying the same approach to the North Polar cap in collaboration with another iMars partner. ACRO products have therefore been processed as well as multi-resolution DTMs from CTX and HiRISE. All the 3D mapping products have been analysed qualitatively by visual inspection and the ACRO processing includes calculating internal quality metrics, which is used to flag bad products.
An automated data mining algorithm was developed to find scene fragments where single or multiple instances of change are detected. This used supervised classification initially with planetary science labelled inputs but will in the near future employ mass public participation from a shortly-to-be-launched citizen science programme under the auspices of Zooniverse. A great deal of effort was placed on determining the optimum Human Factors for incentivising and motivating such participants. Each additional set of a minimum of 10 identifications and notations will be further employed to improve the classification (standing it is believed at around 50%) and the data mining applied to new areas by non-EU funded students at Masters and PhD level. The key to public dissemination is the iMars webGIS system which has been developed to allow experts and members of the public to examine different parts of the planet for changes as well as perform geomorphological and geological research. Dissemination of the underlying products will take place through ESA PSA (for HRSC products) and NASA PDS (for NASA ACRO and CASP-GO 3D mapping products).
The iMars base data can be used by the ESA ExoMars Trace Gas Orbiter 2016 and subsequent ESA missions to provide the necessary inputs for the selection of a future landing site for the ESA ExoMars 2020 rover and for any Mars Sample Return missions in the 2020s.
It will greatly extend the use of the archived data by providing mapped and co-registered images.
Project Context and Objectives:
Context and Objectives
Almost fifty years have elapsed since the NASA Mariner 4 first took the first pictures of the Martian surface. Over that time, the resolution and quality of these images has improved from tens of kilometres down to 25 cm. Over the intervening ≈50 years, many areas on Mars have been repeatedly imaged as each new science team has “re-invented the wheel” and gone back to the same places highlighted by these early images. Historically, the driver for these scientific studies has been the “hunt for water” but more recently, there is an increasing interest in different dynamic phenomena associated with motion of dust and possibly different permafrost phenomena. The revolution in planetary surface observations, especially in 3D imaging of surface shape, has led to the ability to be able to overlay different time epochs back to the mid 1970’s, to examine time-varying changes, such as the recent discovery on Mars of mass (e.g. boulder) movement, tracking inter-year seasonal changes and looking for fresh craters from meteoritic strikes. As our exploration of surface changes proceeded we determined that the polar regions had some of the most significant surface changes and so a great deal of emphasis was changed to focus on these regions.
In iMars, we wanted to mitigate the deleterious effects of increasing knowledge and accuracy of where the spacecraft was located in its orbit and where the cameras were pointed to allow fully automated computer algorithms to eliminate the impact of these errors and provide co-registered image products so we can playback any area on Mars, which had repeat coverage. Of course, in many cases no change can be observed, however surprisingly the surface is much more dynamic than at first thought particularly with regard to very dynamic phenomena like dust devils. When the technology developed by iMars is rolled out across the planet in future it is very likely that significant new discoveries will be made. For polar regions, the number of repeats is significant as well as for regions such as the landing sites, Vallis Marineris, the Tharsis volcanic region and other regions repeatedly imaged on the boundary of the so-called dichotomy. Little, if any, information is available on the variability of the Martian surface, although the annual and seasonal changes associated with the polar cap expansion and contraction and the rapid spread of dust storms has been relatively well monitored in the past. However, the Martian surface is undergoing constant changes from our observations of just 3% of the surface so we do not yet know what we can fully expect to find.
Objectives (taken from the original DoW):
• To characterize the nature and characteristics of HRSC sensor orientation variations within Mars orbit.
• To assess suitable techniques for extracting height control information from MOLA altimetry.
• To compile a contiguous HRSC DTM data product by adjustment and integration of stereo-image coverage on multiple overlapping image tracks on global scale and assess the performance achieved.
• To show the impact of multi-orbit integration on an improvement in DTM mosaicing.
• To integrate the HRSC DTM data product generated in Task 3.2 into the global project data base.
• To mosaic the resultant DTMs with existing MOLA profile data to ensure the best possible global DTM. Surface matching to be employed rather than rigorous block adjustment.
• To generate the most complete CTX stereo-derived DTMs and ORIs for much of the Martian surface and employ the automated co-registration software to ensure that these images are co-registered to the HRSC ORIs. To generate a set of ORIs for single view CTX images using the best DTM.
• To generate the most complete HiRISE stereo-derived DTMs and ORIs feasible and employ the automated co-registration software to ensure good co-registration with CTX.
• To apply the automated co-registration software using the HRSC-CTX-HiRISE ORIs to all prior NASA Mars imagery from Mars Global Surveyor, Mars Odyssey and Viking Orbiter back to 1976. The results of each feature-pair match will be fed into updating the SPICE kernels for each historical image and subsequent orthorectification using the global DTM.
• To ensure that all the resultant images are entered into the web-GIS database and can be accessed directly within existing GIS such as ARC® and GRASS.
• To generate a Design Document (DD) consisting of functional specifications and implementation approaches for a planetary Mars-based webGIS (hardware), DBMS and GIS service specifications, user access specifications, and specifications for integration of software modules defined through a common architecture API (software).
• To setup a development and testing/validation webGIS core-system environment.
• To implement modules developed by consortium.
• To setup an environment (as specified in the DD) for open as well as credential-based access by researchers and general public.
• To provide access and means for validation and stress-testing of the webGIS and, by that, demonstrate operability.
• To investigate the use of co-registered datasets and available matching tools for change detection.
• To explore different change detection methods applied to time series images of the Martian surface.
• To choose the best possible method to flag images where change has occurred including time-lapse sequences to assess whether such a sequence is a true change or just an error of commission.
• To provide data to WP7 for validation using crowd sourcing.
• Review human factors research on human computation & automation.
• Test HF theory in the domain of Martian geomorphological feature recognition.
• Build a MarsZoo project to produce data on a range of features, taking into account the results of HF testing.
• Validate MarsZoo results with scientific users.
• Communicate “best practice” results & guidelines.
• To coordinate dissemination activities among all project partners and ensure visibility of project aims and results to science community and general public.
• To develop strategies for further exploitation of the project results and identify opportunities for follow-up projects.
• To implement efficient communication with scientific community and general public for enhancing and promoting the use and benefit of iMARS tools and data.
• To employ the NASA Regional Planetary Imaging Facilities (RPIFs) to promote the products and tools developed by iMars.
• Ensure that adequate technical and administrative management procedures are established and implemented.
• Report on project status and achievements.
• Implement a policy leading to the minimisation of project risks via timely identification of problem areas and their resolution.
• Perform timely and correct financial project management.
Project Results:
Description of Main Results
The iMars project focused on developing tools and value-added datasets to massively increase the exploitation of space-based data from NASA and ESA mission imaging and 3D data beyond the PI teams. iMars has successfully created substantially more complete and fused 3D models of the surface from combined stereo and laser altimetry and used these 3D models to create a set of co-registered imaging data through time, permitting a much more comprehensive interpretation of the Martian surface to be made by scientists in the future. Emphasis has been placed on co-registration of multiple datasets from different space agencies and orbiting platforms around Mars and their synergistic use to discover what surface changes have occurred since NASA’s Viking Orbiter spacecraft in the mid-1970’s. iMars has brought together the best expertise in Europe for the processing of Martian orbital data within a single environment for handling, visualising and interpreting these data. The ESA Mars Express High Resolution Camera (HRSC) has been used as base data, where possible as its fundamental image resolution of 12m and DTM resolution (from 50-150m) makes it ideally suited to processing NASA imagery of resolution from 6m-100m.
The iMars datasets/products can be used to provide the necessary inputs for any Mars Sample Return missions in the 2020s. It greatly extends the potential use of archived data by providing mapped and co-registered images. The resultant time-stamped imagery has been interfaced to automated data mining analysis software based on techniques developed for Earth surveillance. We have also built on the huge momentum, developed in the Zoouniverse system by building a “Mars in Motion” project for mass public participation in the feature mapping of Mars. Successful co-operation with US colleagues through the Technical Advisory board at annual project meetings, as well as participation from European scientists through several workshops have brought the project to a greater level of scientific interest and a public event at the end of the project have helped to promote the project to a much wider audience.
WP1 Software for Automated DTM/ORI extraction from CTX/HiRISE data
In WP1, UoS and EPFL developed a stereo processing system while UCL assessed existing processing chains from NASA and USGS and compared the results from UoS and EPFL with these for both accuracy, completeness and speed. Results were then compared against each other and with the HRSC mapped products to find a suitable processing system for large-scale processing.
UoS modified an automated stereo processing pipeline developed by Kim and Muller (2009) for a more generic planetary stereo processing into an automated DTM pipeline and advanced sensor-modelling system. Consequently due to a successful “coarse to fine” control strategy, stable horizontal and vertical photogrammetric accuracy of the resultant iMars DTMs was achieved when compared with a MOLA DTM. The UoS algorithms were successfully applied to produce 12m CTX and 0.75m HiRISE grid spacing Digital Terrain Models (DTMs) and ortho-rectified images (ORIs) after application to CTX and HiRISE stereo images.
The major differences to the original algorithm published by Kim and Muller (2009) for the UoS stereo processor are:
1) Employing the ISIS “cub” level 1 image matching software.
2) Requirement for only a few 3rd party software libraries. Minimising unnecessary components aiming to create a certain quality DTMs with only essential modules.
3) Minimising parameter dependency during the stereo matching process.
4) Final system consists of C++ execution files and a Python wrapper.
5) All main execution files were designed with openMP functionality for high speed parallel processing on the MSSL linux cluster.
The UoS software delivered to UCL consisted of the following components
1) Conversion routines to permit a generic sensor model to be generated from interior & exterior orientation parameters.
2) Efficient stereo image matcher which can manipulate both texture-less and high slope topographies together
3) Bundle block adjustment along with space resection software routines
4) Space intersection routines
5) Other auxiliary routines to handle topographic product
EPFL had previously developed an additional processing pipeline for high resolution DTMs. It employs stereo matching methods and approaches utilised for MOC stereo data (Ivanov, 2003; Ivanov & Lorre, 2002) in conjunction with radiometric and geometric image processing in ISIS3. This technique is capable of deriving tie-point co-registration at sub-pixel precision and has proven itself when used for Pathfinder and Mars Exploration Rover, MER operations (Lavoie et al., 1999).
The basis for image correlation is the Gruen algorithm (Gruen, 1985), which was originally employed in the amoeba optimization method (Nedler and Mead, 1965). The method is an iterative approach. Given template image and search area, it searches for the best match in the search area. This search can take a considerable amount of time and therefore this part of the code is the prime candidate for implementation on parallel architecture.
The next step is performed using the ISIS3 Application program Interface (API). The ISIS3 API is used to derive ray information for ray-ray intersections for pixels identified by image correlation. Once ray intersections have been calculated they give precise XYZ information in a Mars Reference frame. XYZ values are then converted into triplets of (Latitude, Longitude, Radius). The final product is created by triangulation and gridding of this dataset. UoS utilise Generic Mapping Tools (GMT) to produce regularly spaced grids, using interpolation in places where we have found no correlation. Filtering bad correlation points is an important step for the final quality of the DEM and this step still has to be tuned for each stereo-pair in order to achieve the best possible DTM quality.
UCL applied the NASA AMES stereo pipeline (Moratto et al., 2010) “as is” as well as the USGS stereo processing system based on the use of ISIS software for creation of images from the original level-1 HiRISE separate pushbroom files and the COTS SOCET® system for the DTM and ISI for the ORI production. The UoS software was ported to UCL but good results were not achieved.
The following products were produced for the WP1 assessment by EPFL, UoS and UCL:
CTX and HiRISE DTM and ORIs co-registered to HRSC DTM & ORI for the selected data sets for the sites covering three NASA Mars rover systems: MSL-Curiosity, MER-A Spirit and MER-B Opportunity sites with the following specifications:
• Grid spacing (hereafter called resolution) for CTX DTMs (20m) and CTX ORIs (6m);
• Resolution for HiRISE DTMs (0.75m) and ORIs (O.25m);
• Output data format: geotiff with separate world files.
A pairwise comparison of any set of algorithms as described above was performed for the same regions on Mars and compared against corresponding HRSC DTMs of lower resolution.
For this purpose we chose to perform an evaluation of CTX images of the area surrounding the three most observed rover sites. The supporting DTMs and images (for selecting geodetic GCPs) were taken from the HRSC products overlapping with the CTX images. Subsequently, some HiRISE stereo-pairs overlapping with the selected CTX images were also processed using SOCET® and the NASA-ASP system for comparison purposes.
The average and standard deviation of the differences between the HRSC and ASP, SS, and UoS DTMs for CTX was +1.4 ± 84.2m −2.1 ± 84.4m and −2.7 ± 84.9m respectively. The same differences for the HiRISE instrument were −13.3 ± 19.7m +4.2± 19.7m and +2.3 ± 37.2m. The large dispersion of the differences is due to a larger number of surface features over a larger area for CTX and a smaller number of features for a smaller area covered by the HiRISE instrument. The results were considered to require further improvement in global consistency, completeness and robustness in WP4.
In a further validation part of WP1, DLR found large height offsets between WP1 DTM results and MOLA/HRSC and could relate this to the height datum definition in the USGS software system (ISIS). DLR DTM validation revealed a miss-alignment of the default MOLA data set of ISIS with respect to the reference PDS data set. Further analysis on this issue revealed fundamental inconsistencies in the GDAL and ISIS routines for HRSC import as well. Data conversion procedures strictly applying the written definitions for the respective software systems and data products have been set up in order to be able to validate the CTX/HiRISE data products. The latest version of GDAL took these reported issues into account to address future issues. D1.1 and D1.2 were produced for this WP.
WP2 Software for Auto-coregistration of NASA images to HRSC ORI/DTMs
UCL developed an automated algorithm for the co-registration of feature points extracted from overlapping NASA images at level-1 called EDR (Experimental Data Records) to ESA-DLR-FUB HRSC mapped data. This automated co-registration and orthorectification (ACRO) system has a number of novel features, including the use of SIFT features, coupled decomposition for sub-image creation for massive >1Gb sized images, detection of sub-pixel features in HRSC and corresponding correlated features prior and future non-HRSC (i.e. NASA) images and creation of a set of latitude/longitude/height points in HRSC and overlapping NASA images which can be employed in future to update the SPICE kernels for these NASA images as well as be employed as geodetic control points for HiRISE, CTX, CaSSiS and serendipitous stereo from other NASA sensors.
Cameras Years Resolution (m) No. images Data
Viking Orbiter 1 1976-1980 8-1800 ~32 000 29G
Viking Orbiter 2 1976-1978 8-1800 ~15 000 29G
MGS MOC-NA 1997-2006 1.5-12 97 097 ~350G
MO THERMIS 2002- 18-36 ~205 000 ~6T
MeX HRSC 2004- 12.5-25 ~5 000 (nadir) ~3T
MRO CTX 2006- 6 ~79 000 ~12T
MRO HiRISE 2006- 0.25-0.5 ~90 000 ~120T
Table 1: Summary of High-resolution (≤100m) Mars image data (aside form VO). Note that HiRISE acquires the same data volume every day that 2 VO missions acquired over 4 years.
Table 1 summarizes the numbers of orbiter images and associated data volume acquired to date. This large quantity of data was employed to generate repeat coverage maps of the Martian surface [see Figure 1: High-Resolution Mars REPEAT Coverage taken from Sidiropoulos & Muller 2015], thus making it possible to assess where surface changes may be able to be tracked over the planet. Table 1 shows that, even if images with resolution finer than 20 metres per pixel are only taken into account, change detection is theoretically possible over a significant part of the Martian surface. For example, the area mapped more than once during (Northern Hemisphere) spring is larger than the entire area of Asia, while the area mapped at least thrice in any season is on the order of an Earth’s continental area.
The UCL ACRO software generates orthorectified images of the Martian surface from orbital data of NASA missions to Mars (VO, MOC-NA, THEMIS, CTX and HiRISE instruments) co-registered to HRSC orthorectified images and orthorectified using HRSC DTMs. See Sidiropoulos & Muller (2017) and deliverables D2.1 & D2.2 for further details of the algorithm and software.
In parallel, UoS conducted co-registration of their target area images and DTM products over HRSC employing their in-house co-registration routine. The main purpose of UoS efforts are the establishment of ground control point setting for epipolar rectification and geodetic control for the WP4 stereo processing. In order to achieve fine resolution and high accuracy geodetic control with less than few horizontal pixels and 10m vertical deviations, a robust geodetic control method was implemented combining image-to-image correlations. The algorithm is based on a combination of machine vision approaches consisting of feature point establishment using Scale Invariant Feature Transformation (SIFT), a variation of an optical flow algorithm, least squares correlated refinement and pyramidal reorganization employing affine transformation. UoS also implemented an image pointing adjustment method using rigorous and non-rigorous sensor models, which is then applied to co-registration of stereo photogrammetric products. UoS studied and tested involved procedures using their in-house software and a combination of some subroutines from the open source USGS ISIS.
WP3 Bundle block-adjusted DTM mosaic from HRSC
The iMars project is developing a user platform for Mars surface change science, consisting of a collection of co-registered and terrain corrected data products from Mars orbital imaging data sets, and specific tools for producing, exploring and analysing change shown in these (data mining, webGIS, crowd sourcing). For GIS analysis, and in particular for change detection, co-registration of all involved datasets to a common geometric reference is essential. The primary reference data sets used by iMars for this purpose are the digital terrain models and orthoimages derived from HRSC map products. Therefore availability of these data products together with a detailed assessment of their geometric characteristics has been a major cornerstone within the project.
The technical reasons for this are intrinsic difficulties associated with co-registering images to a reference dataset represented by an elevation model such as the NASA Mars Orbiter Laser Altimeter experiment (MOLA), such that for the case of the highest resolution data sets such as data of the MOC narrow angle imaging device of Mars Global Surveyor or the HiRISE camera of Mars Reconnaissance Orbiter (MRO), a geometric relationship with the MOLA reference usually cannot be established directly due to the huge differences in spatial resolution. Conversely, HRSC, as a stereo imaging sensor and with imaging at a resolution of 12 m, provides data that can demonstrably be precisely registered to MOLA heights by means of photogrammetric adjustment, so that HRSC terrain-corrected images [see Figure 2: Colour-coded shaded relief maps of the HRSC multi-orbit DTMs for Mars Quadrangles MC-11-E (Oxia Palus, East) and MC-11-W (Oxia Palus, West). E-W extent is about 2663 km (at the equator), N-S extent about 1780 km, grid spacing of 50 m.] can in turn be used as a basis for co-registering other image data.
One of the major objectives of WP3 consisted in providing the necessary specifications concerning HRSC multi-orbit data products as a new base data set, in particular their use as a reference dataset. In addition, a review of relevant methodological background is provided as well as a summary of specific investigations in the areas of height control and modelling of image orientation that were intended to provide useful guidance for optimising DTM production. The dataset specifications were discussed in the context of previous topographic Mars data products and current usage trends for such data products. The compiled set of product specifications is available to the Mars Express HRSC team for adaption to the systematic generation of HRSC multi-orbit products.
The suggested parameters specifying the multi orbit data products were obtained using processing tests concerning the representation of detail in the DEM, the internal geometric consistency between strips, and the consistency with external reference heights.
Furthermore, we analysed the original HRSC mission data record with respect to important parameters such as image resolution, we reviewed and analysed existing results of single-strip product generation by the HRSC Team, performed image mosaicing tests that were evaluated in terms of brightness homogeneity and the effectiveness of image edge elimination, and, finally, reviewed relevant PDS standards. The analysis of the methods for height control was based on processing results obtained using two different tools, bundle adjustment and a procedure for trend surface analysis and correction. The results provided different implications for the suggested set-up of the product generation, most importantly a recommendation to apply bundle adjustment as the general 3D adjustment method in combination with a final trend surface correction for height. The analysis of Mars Express orientation data properties was based on analysis of existing results from HRSC single-strip processing, comparison with available high sample rate orientation data from ESA, and star observations by the SRC camera. The results showed that a tie-point sampling rate of 4 seconds is generally required to cover typical MEX spacecraft oscillations, and 2 seconds are recommended considering the finite precision of tie-point measurement.
The second main focus of WP3 was devoted to the geometric validation of HRSC multi-orbit DTM products, which is a very relevant aspect concerning the further use of such data as a geometric reference dataset. Two DTM prototype datasets were used for validation (MC-11E and MC-11W). The geometric quality of the results has been assessed with respect to internal parameters such as the mean 3D intersection error as well as external data, i.e. MOLA altimeter heights. The results were rated very encouraging because they confirmed that the already high and well-controlled internal precision of the single-strip 3D models produced and released to the PDS/PSA archives for several years are preserved throughout the new integrated bundle block adjustment process. We also investigated the quality of co-registration of adjacent HRSC strips before and after block adjustment and could show that residual offsets are reduced by a factor of 3 to 4 for the horizontal coordinates, or to about one pixel for the nadir images, which allows the production of panchromatic orthoimage mosaics with the highest available resolution. A comparison of the results of our new approach to multi-orbit processing with simple mosaics of local DTM patches or strips demonstrated that typical obstacles to the successful application of the latter can be efficiently avoided by the new approach. This refers to avoidance of edge artefacts, related to weakly constrained interpolation close to DTM borders, masking of higher resolution datasets by lower resolution datasets, and increased coverage by filling of data gaps present in some of the datasets.
WP4 Global DTM/ORI production & validation
The HRSC data products in the iMars data base include more than 1402 single strip PDS HRSC DTMs and orthoimages produced by DLR, as well as the multi-orbit products for the MC-11E and MC-11W tiles used for validation in WP3 which themselves comprise some 200 single-strips. At a recent HRSC CoI meeting held just after the HRSC CoI meeting, DLR announced that out of 16,788 orbits performed by the ESA Mars Express mission, some 5,000 contained HRSC data with 75.9% having nadir images with <20m resolution and L4 products being available up to orbit 8,000 within the HRSC CoI team and shortly through PSA/PDS to the scientific community. It appears that around one quarter have been processed to date with DLR recently re-commencing the production of single-strip bundle adjusted L4 products at a rate of several tens per month.
DLR produced Lambertian BRDF-corrected, block adjusted orthoimages in 5 filters over the entire MC-11W and MC-11E areas. These orthoimages were brightness-adjusted and mosaiced using a global set of reflectance standards generated from the Mars Global Surveyor, TES instrument. In addition, all the data from Mars Express orbits between 2004 and 2016 covering the MC-11E and MC-11W areas were processed to Level-4 single strip data products (DTMs and orthoimages) using the new bundle-block-adjusted orientation data. These represent a preliminary version of new, updated Level-4 products to be derived from bundle block solutions in the future. For this reason the data sets are provided for internal use by the consortium in the frame of iMars and will be replaced in the near future by an official version released to PSA/PDS.
In addition, 39 HRSC orbits were processed by UCL with EXTORI files from FUB to produce complete coverage of the South Polar Residual Cap both with 3D Digital Terrain Model (DTM) and a set of orthoRectified images (ORIs) shown in Figure 3 [Base SPRC DTMs produced by UCL and FUB], and, Figure 4 [Orthorectified Nadir images using base SPRC DTMs]. See D4.1 report for further details.
In WP4, a fully automated multi-resolution DTM processing chain was developed by UCL for NASA CTX and HiRISE stereo-pairs, called the Co-registration ASP-Gotcha Optimised (CASP-GO), based on the open source NASA Ames Stereo Pipeline (ASP) (Moratto et al., 2010; Broxton et al., 2008), tie-point based multi-resolution image co-registration (Sidiropoulos & Muller, 2015), and Gotcha (Shin & Muller, 2012) sub-pixel refinement method. The implemented system guarantees global geo-referencing compliance with respect to High Resolution Stereo Colour imaging (HRSC), and hence to the Mars Orbiter Laser Altimeter (MOLA), providing refined stereo matching completeness and accuracy from the ASP normalised cross-correlation.
The CASP-GO pipeline for both sensors is shown in Figure 5a: Flow diagram of UCL-Ames CASP-GO processing chain for CTX, and, Figure 5b: Flow diagram of UCL-Ames CASP-GO processing chain for HiRISE. Apart from the ASP pre-processing, cross-correlation matching, triangulation, and DTM/ORI generation, five additional workflows are introduced to further improve the ASP results. These include (a) a fast Maximum likelihood sub-pixel refinement method to build a floating-point initial disparity map; (b) an outlier rejection and erosion scheme to define and eliminate mis-matches; (c) an ALSC and region growing (Gotcha) based refinement and densification method to refine the disparity value and match un-matched and/or mis-matched area; (d) co-kriging grid-point interpolation to generate the final DTM as well as height uncertainties for each DTM point; (e) ORI co-registration w.r.t. HRSC.
Based upon the “cleaned-up” sub-pixel disparity map, an Adaptive Least Squares Correlation (ALSC) refinement is performed on all the remaining disparity values iteratively. These refined disparity values are used as seed points for Gotcha (Grün-Otto-Chau) densification (Shin & Muller, 2012). The Gotcha matcher is based on ALSC and region growing. It is very robust and accurate, but rather slow for large-scale image matching since it tries to match every point iteratively and re-sort all seed points according to a “best first” strategy when a new patch is matched. However, given sufficient number of sub-pixel disparities that pass the outlier rejection schemes, small difficult regions can be matched with Gotcha accurately. For example, the geometrical distortion generated by different viewing angles can be addressed with Gotcha by modifying the shape of the ALSC window iteratively, albeit currently only with a parallelepiped using an affine transformation for the distortion model.
With Gotcha densification, we can achieve improved completeness for the final DTM without significantly smoothing out sharp features from a large matching kernel. Generally, a larger ALSC window and higher maximum eigenvalue yields better completeness in disparity and hence for any resultant DTM. The similarity values from the Gotcha matcher are then used together with camera model intersection error and co-kriging parameters to produce DTM uncertainty values per gridpoint.
In CASP-GO, UCL added a Mutual Shape Adapted Scale Invariant Feature Transform (MSA-SIFT) based co-registration workflow to the final ORI and DTM product. Technical details of MSA-SIFT and the evaluation on ORI/DTM co-registration results for MER and MSL are described in (Tao & Muller, 2015). In this ORI co-registration and DTM adjustment work, we take HRSC ORI as the reference image for CTX ORI co-registration and subsequent shift of the corresponding CTX DTM according to the CTX ORI to HRSC ORI transformation. For the HiRISE co-registration, we take already co-registered CTX ORI (to HRSC ORI) as the reference image for HiRISE ORI co-registration and subsequent transformation of the HiRISE DTM to ensure multiscale congruency.
UCL mirrored the HRSC, CTX, HiRISE PDS data volumes from JPL in a local shared storage system in order to speed up the production process with an option such that if data is unreachable it can be read from the original source again. At UCL-MSSL, the developed software is installed in a shared directory, which is accessible from 14 Linux processing blades (10 with 16 cores and 48GB RAM; 4 with 24 cores and 96GB RAM). Jobs are controlled via a local desktop machine and distributed to the 14 processing blades with multiple sessions of multi-threaded processing. Processed results are stored in several 1TB RAID storage disk partitions and logged back to the local controlling desktop. Failed jobs can be examined through detailed log files and could be reprocessed automatically with different processing parameters in the future.
In the meantime, UCL was successfully awarded with free access to $20,000 of computing resources from Microsoft® Azure Cloud for Research. UCL worked on the virtual machine set-up, software integration, and test processing on the Microsoft® Azure cloud computing since early 06/2016 and started batch processing of 1540 CTX stereo pairs (the original published list of CTX stereo-pairs from the NASA Ames group published online in 2012) in 07/2016 [see Figure 6: Location of all 1540 18m CTX DTMs processed by the end of 2016. Where no level-4 HRSC DTM exists (red areas), the CTX products are not co-registered to a global co-ordinate system] which completed in mid 12/2016. UCL was later awarded an additional $20,000 computing resources from Microsoft® Azure Cloud for Research in early 01/2017 for finishing up the rest of the ≈4,000 CTX stereo pairs [see Figure 7: Locations of all of the 18m CTX DTMs processed by the Microsoft Azure® including overlaps with those shown above. It is expected that this larger dataset will be completed by mid to end April 2017 after project ends], some of which overlap with the previous set. This new set was derived using a novel stereo intersection algorithm based on the one described by Sidiropoulos & Muller (2015). In all cases only stereo-pairs with 100% overlap were selected.
The CASP-GO processing chain has been tested/applied to stereo Mars Reconnaissance Orbiter (MRO) Context Camera (CTX) imagery (6m) over the Mars Exploration Rover (MER-A, B), Mars Science Laboratory (MSL) and a large area mosaic over the US Geological Survey’s MC11-E/W area (~100 stereo pairs) and then applied to the production of planet-wide DTMs and ORIs from CTX (~1700+3800 stereo pairs) and is being applied MRO High Resolution Image Science Experiment (HiRISE) 25cm NASA images (~11 stereo pairs).
The overall statistics for CTX are 1915 pairs have HRSC out of all of the total of ≈5,300. Out of 1540, 882 pairs don't have HRSC. Out of 3963, 2491 pairs don't have HRSC. However, for the Polar Regions, there is now a complete set of level-4 data so for [see Figure 8: North Pole (a & b) and South Pole (c & d) coverage of CTX in the original 1540 set (left column) and 4000 set (right column). N.B. Please note different Colour key meaning here], these figures are slightly misleading. A large percentage of the processed DTMs have been assessed using a 5 star rating scheme by UCL, i.e. 1 – failed, 2 – major problem, 3 – minor problem, 4 – good quality, 5 – very good quality. 503 out of 620 DTMs have been rated as 3+, i.e. 316 good and very good, 187 have minor problem. Only 31 out of 620 failed all due to bad input images (≈5%).
A geometric quality assessment for the CTX DTMs produced within iMars was performed over 3 test sites (the landing sites of the MER-A, MER-B, and MSL rovers) by DLR. The validation methods included proven techniques for visual assessments based on color-coded height displays, shaded relief maps, and contour lines. In addition, quantitative measures using height statistics, probability of crater detection (to assess effective resolution), and comparison with similar results for existing DTM datasets were applied. A similar set of validation techniques was applied to the HiRISE DTM results over the same three test sites (MER-A, MER-B, MSL landing sites). See D4.5
HiRISE images are 10 to 15 times larger than corresponding CTX images (see further details in D4.3) and contain much richer fine scale features at 25cm that may vary between stereo images. The HiRISE EDR products also contain systematic noise such as strip noise and gain variations. Based on the results from the experimental sites of MER-A, MER-B and MSL, several modifications were made to the original ASP DTM processing chain in order to produce best results for HiRISE images.
A large number of HiRISE DTMs are still being processed in the AWS® cloud computing VMs. These include 441 stereo pairs for the areas that have 5 or more repeat HiRISE observations. This is still on-going due to the large computing resource required and limitations from the free AWS® credit. Once they are completed, the 441 HiRISE DTMs will be integrated into the iMars webGIS together with the existing UoA processed HiRISE DTMs. The global coverage of the UCL HiRISE DTMs that are being processed and the UoA HiRISE DTMs are shown in Figure 9 [Global coverage of the UCL HiRISE DTMs that are being processed and the existing UoA HiRISE DTMs]. Figure 9a shows Global coverage of the UCL HiRISE DTMs being processed and the UoA HiRISE DTMs available to date in Equirectangular projection system. Figure 9b shows the North Pole and Figure 9c shows SPRC, both in Polar Stereographic projection. Purple for the SPRC have existing CTX DTMs].
In parallel, UoS re-implemented WP1 software into a more scientific oriented form especially considering geodetic control and point density, which are essential properties for geological/geomorphological analyses. Together with the UoS stereo software and manual parameter setting, CTX/HiRISE DTMs over scientifically interested areas have been processed and delivered. UoS conducted their own validation work for some CTX and HiRISE products over a few places using mainly MOLA track profiles. This assessment procedure gave confidence for further scientific investigation using UoS DTMs and orthoimages.
Target areas for the application of UoS stereo processors were established in two categories. The processed data sets including DTM and orthoimages over the following areas (> 20 sites):
• Area oriented targets over Naktong valley, Chinju crater, Naju crater
• Feature oriented target areas such as Lobate Debris Apron (LDA) over Euripus Mons, fluvial channel over Baharm and Elysium, tectonic feature around Claritas fossae, Inter-crater features over Elysium, Mojave and Jazero [see Figure 10: Target area locations of CTX and HiRISE DTMs from the UoS stereo processor].
UoS showed that several scientific studies employing UoS processed DTMs are actively undergoing and will be continued in future.
The other main task of WP4 is about the co-registration and orthorectification of NASA imagery to the HRSC baseline, a task that demands not only the development of a core efficient algorithm that can achieve a fast and reliable co-registration of high-resolution multi-instrument planetary images to a common baseline but also a number of variations that overcome the apparently random and incomplete nature of the current high-resolution Mars imagery. These variations were a task that required substantial human resources to be applied. This created a bottleneck in the processing of the global imagery, which while being under way is not expected to be completed before the end of the project. However, a substantial number of images will be processed, and the pipelines for extending this to the rest of Mars orbital imagery are just a matter of computer and human resources, especially since the developed algorithm is fully automatic and requires minimum human resources. Note that UCL has already secured 20,000 core hours of funding from Microsoft Azure® which may be used to process the rest of Mars orbital high-resolution imagery, before January 2018.
Moreover, even though the developed algorithm achieved state-of-the-art resilience to resolution differences between the input image and the baseline, there is a limit on the resolution differences for co-registration in order for it to have any practical meaning. For example, a HiRISE image has a resolution of 25cm/pixel, is 50 times finer than HRSC resolution. This means that a 50x50 HiRISE patch corresponds to a single HRSC pixel. The matching of images with such large resolution difference would be possible if (according to Shannon’s sampling theorem) the spatial frequency of the terrain is such that the sampling difference wouldn’t change the signal (i.e. the image). In practical terms, this would mean features of size no less than 25 metres (since 12.5 m/pixel is HRSC resolution). On Mars, the surface is very rarely so “simple”, since high-resolution imagery both from orbit and from rovers have revealed features of great detail, on scales of centimetres or even millimetres. Under this circumstances, co-registration of images with so much resolution difference is a very challenging task, especially when following the design principle to build a fully automatic pipeline that can process large volumes of data without requiring any parameter tuning. The systematic, batch-mode, co-registration of products with such high-resolution should wait for a 3D baseline of higher resolution (between HiRISE and HRSC), before they can be added to the dataset of Mars geometrical aligned image dataset. CTX is being used for this purpose in future.
Six different variations of ACRO pipeline were developed and applied and are described in D4.4. A composite of 2 CTX ACRO’d images with HRSC is shown in Figure 11: A mosaic of 2 CTX images and an HRSC ORI. The CTX images were co-registered to the HRSC ORI using ACRO v2.2. The yellow-coloured image in the top-left part of the figure is CTX image D14_032511_0959_XI_84S078W_ORI (Ls 345) while the blue-coloured image in the centre is the CTX image B11_013813_0955_XN_84S078W (Ls 299). The background image is the level-4 HRSC ORI H2288_0000 (Ls 312). All images are shown using a polar stereographic projection.
WP5 iMars web-GIS Software
iMars created a repository of raster data from ESA and NASA missions covering the last four decades of Mars orbital exploration. With these data, systematic surface change surveys can be performed using auto co-registration and data mining techniques. This will lead to a better understanding of surface processes and relative ages of different areas.
Each dataset is inherently associated with its imaging acquisition time. Thus, a stack of co-registered images will allow navigation through time layers using time sliders and associated techniques. In order to make these data accessible to public users as well as researchers, a webGIS platform was created by FUB, which offers interactive navigation and dedicated visualisation facilities. It builds upon a spatial database management system and an open-source Web Map Service provided by an open-source webserver backend.
Within WP5, FUB has developed a planetary web-GIS able to handle massive amounts of data and various different types of data layers. We have achieved our goals for an intuitive interface with a high performance display of the different vector and raster data types. An overview of the overall client-server based system design is shown in Figure 12 [System design of the iMars webGIS consisting of the frontend (including a regular web server) and the backend with its several storage subsystems and processing components.].
The system consists of a frontend part, which is essentially the web page, and JavaScript files, which are loaded by the user’s browser. They are directly located on a webserver. The backend parts get loaded indirectly from the browser and consist of a single or possibly multi-server setup of a web server with an attached tile cache and a web server with attached raw data storage. The database system can be located on a separate system for load balancing purposes. The load balancing design enables performance scaling for high user demands.
The user interface as part of the frontend in the browser consists of several modules built upon the open-source OpenLayers library. We have developed additional control panels to interact with the map and its contents. Figure 13 [Overview of the existing and newly developed graphical user interface elements] gives an overview of the available components. In particular, the newly developed components are:
1. Modified Layerswitcher module (derived from the existing ol3-layerswitcher project on Github). New functionalities such as group folding/unfolding, group selection, transparency sliders, information buttons, layer sorting and manual re-ordering and layer deletion have been added.
2. Toolbar for the selection of single images from the respective footprint coverages and for the creation of dynamic layers for each of the single co-registered image. A play, pause and stop button group is available for iterating over a time series of images.
3. Time panel for the filtering of the available coverage data for time. Both universal time and Martian time fields are available for the start and stop time entry. Additionally, the time span can be selected with a time slider.
4. Projection switcher, a small element to choose between the three different projections for global equidistant cylindrical (equirectangular), south polar stereographic and north polar stereographic.
5. Goto lat/lon is a two-field entry mask for entering individual latitude/longitude coordinates – the map centre moves to the entered values upon execution.
The remaining controls in Figure 13 are regular Openlayers controls for full-screen toggling and zooming functionality.
The work in WP5 consisted also in the assembly of various data layers for visualisation in the webGIS and in GIS services. In particular, the webGIS offers the following data layers (bottom to top in the webGIS):
1. A combined topographic group layer consisting of several layers of DTMs of different instruments (HRSC, CTX, HiRISE) as the result of work packages WP1, WP3 and WP4. The DTMs are displayed as colour-coded hill-shaded maps. The single topographic layers with their different cell sizes integrate seamlessly into each other so that the user can zoom in and out without recognising the transition between different datasets. Despite that the single datasets can be turned on and off as single layers to be able to investigate single instrument’s data products exclusively.
2. A group layer of the HRSC image mosaic products as released by the HRSC team. The group contains the grayscale mosaic of MC11 in 12.5 metres per pixel as well as the colour mosaic in 50m per pixel. Both layers are equipped with transparency sliders to enable the user to view the colour-coded topography information together with the images. This data is currently exclusively visible on the iMars webGIS server.
3. Co-registered ortho-rectified images of CTX and HiRISE from different providers within WP4.
4. Vector layers for generic informational needs, such as a quadrangle overlay to visualise the borders of the quadrangles, a landing sites layer with all the past successful and planned landers on Mars, and a nomenclature layer with the official feature names of the IAU working group for planetary system nomenclature (WGPSN).
5. Footprint layers of the available NASA and ESA image data on Mars connected to the geometric database as described above. The footprint layers allow spatial queries and attribute information via on-screen pop-ups. Within the attributes are links to existing sources of the original datasets for further information and download. The footprints can be filtered using a slider-based selection tool to set start- and end-points in time.
6. Additional footprint layers are available for the existing automatically co-registered single image data created in WP4. These footprint layers serve as a selection database for visualising the available time series of overlapping single images for visualisation of change detection.
7. A colour-coded high-resolution coverage map produced within WP6 representing the available repeat imagery from all available NASA and ESA image instruments for the visualisation of sweet spots for change detection observation.
Figure 14 shows a typical session within the webGIS with selected footprints filtered by a specific time range above the HRSC multi-orbit mosaic from MC11. With the Info button enabled, the user has clicked on a footprint to query attribute information.
Besides this browse and query functionality, the iMars webGIS offers an additional workflow for the visualisation of change detection. It is applied to the data from WP4, the automatically co-registered orthorectified image data, and the HRSC single images. The unique aspect of the tool is the loading of dynamic layers of single images. To be able to control the single image granules from within the frontend with JavaScript, every image has to be loaded as a single layer from the server. The single image loading workflow is controlled via the Toolbar. After enabling layers from within the “orthoimage footprints” group, overlapping geometries can be selected using the “single select” tool. After the selection, the dynamic layers can be added by the “create layers from selection” button. A new group layer called “Orthorectified images” will appear on top of the layer list populated with individual layers for the selected single images. By hovering over the list of layers, the image time will be shown. The user can re-arrange the list manually by drag-and-drop and automatically according to image name, acquisition time or image resolution. The sequence of images can then be played in an alternating cycling loop using the “Play” button (see Figure 15 Snapshot of the visualisation of time series of co-registered single images from the MOC, HRSC, THEMIS and CTX instruments. When the play button is active, the visible image is altered every half second from the list of available images in the group layer). The shown technique proves to be a very good way of visualising surface changes as the difference of features between the alternating images is immediately recognised by the observer.
The large number of images as well as their individual sizes poses a severe challenge to such a system. We have included a dedicated caching system in the pipeline to effectively save image tiles of all types of layers. By the use of image tiles instead of complete images loaded from the WMS, the client will also save the tiles internally which makes the user experience more fluent during panning and zooming. On the server side, we implemented an adapted caching instance to collect and keep the tiles on the server.
After completion of the main targets for the web-GIS functionalities and the ingestion of the products delivered by DLR, UCL and UoS, a performance test of the FUB housed development system was carried out. Performance collection software was used to measure and visualise the simultaneously placed system load of the server machine. It was found that even on the limited developer hardware, the system was usable with ten concurrent users and highly demanding tasks. On the other hand, the application appears as an expert tool with many sophisticated functions which demand high performance. The target group of planetary science expert users is expected to take their time to view the “show-and-tell” YouTube videos and read help pages before or during the use of the software. It should be mentioned in the documentation that the user has to expect long delay times for certain functions. For example, the time of loading many dynamic images may last tens of seconds even on very fast hardware, but the users are “rewarded” with the time-based flickering function afterwards. As long as the end users are well informed with proper documentation, we can expect that they will anticipate and accept delay times whilst working with the application.
An enhanced prototype of the web-GIS was implemented and finalised by the end of 03/2017. A developer environment version of the system with password-protected access was set up (http://imars.planet.fu-berlin.de/dev). Additionally, a public version with password-free access to the internet has been integrated into the consortium’s web page (https://arquivo.pt/wayback/20170603214104/http://www.i-mars.eu/web-gis/) hosted at UCL-MSSL. All the functionality has been tested and proven to perform in a stable manner in a simultaneous multi-user environment.
All the data provided by the project partners were fully incorporated as GIS layers. In addition, the FUB web-GIS also served as an internal data evaluation tool for the main parts of the project with regards to stereo coverage, radiometric image quality, co-registration offsets, processing artefacts and topographic height offsets.
Finally, a QGIS plugin for interactive selection and display of MARSIS and SHARAD radargrams was developed in Python by EPFL and is published in open source (via the GitHub). This operates on multiple OS platforms such as Windows, Linux and MacOS.
WP6 Change detection from Data mining & validation
WP6 is focused on the automatic detection of surface changes on Mars. Change detection processing was conducted using three independent pipelines, developed by UCL, DRL and UoS, respectively. The input data, i.e. high-resolution co-registered images and DTMs processed using the techniques developed in WP1 & 2 and processed in WP4, were processed offline and independently by each partner. Moreover, part of the results of the change detection processing was released by UCL to UNOTT so as to populate their change detection pairs for use within the WP7 crowdsourcing experiment.
A generic automatic change detection approach, developed by UCL, focuses on a batch-mode automatic detection of surface changes on Mars. This pipeline was designed to be context-free, i.e. to detect semantically meaningful changes regardless of the specific type of change. The inputs for this processing pipeline are images, co-registered and orthorectified to HRSC baseline using the UCL ACRO pipeline. As a result, the resolution limitations of the aforementioned HRSC imagery, excluded HiRISE imagery from the co-registered datasets, thus limiting the change detection input to imagery acquired by CTX, HRSC, THEMIS-VIS and MOC-NA instruments with IFOV≥1.5m.
The techniques developed have demonstrated the potential of planetary surface automatic change detection pipelines, thus generating a new family of tools that can assist the understanding of planetary surface dynamic processes. Apart from their clear technological contribution, these techniques have already assisted in the discovery of unreported instances of Mars surface changes, which can be used to augment the catalogues of current day Mars surface activity. In many cases these can be employed to improve numerical GCM models of the Martian atmosphere.
The input of the UCL generic change detection algorithm are pairs of overlapping images, that have been co-registered using the automatic co-registration and orthorectification (ACRO) method that has been developed within the iMars project. For each region, a set of overlapping pairs are selected and pruned to discard image pairs with overlapping regions that are smaller than 256 x 256 pixels.
Subsequently, the preliminary set of input pairs is further processed to extract a temporal series of images, thus eliminating pairs of images that are not temporally adjacent and processing only the minimum number of image pairs. This approach is required to optimise the computational resources without missing any change instances, since by default any change can be temporally narrowed down to a pair of temporally adjacent images (since, if a change can be found between two images I1 and I3 and there is another image I2 that was taken after I1 and before I3 then the change can be found either between I1 and I2 or between I2 and I3). Temporal ordering is particularly useful in areas that have been mapped multiple times, since it reduces the computational complexity from quadratic O(N2) to linear O(N), where N is the total number of images mapping the area. A schematic diagram of the UCL change detection pipeline is shown in Figure 16.
Every remaining pair is fed into the change detection pipeline to automatically identify context-free semantically meaningful changes. Each corresponding HRSC DTM (used during the ACRO pipeline for orthorectification) is also fed in the input, so as to assist in the detection of non-informative changes that are caused by differences in the illumination conditions resulting in changes in the appearance of the surface. Before this step, the input images are initially compared so as to estimate pixels with significantly different pixel values. Because automatic change detection software can estimate large-size changes or medium-size changes, but not changes that take place in a very limited area, changes that happen within “isolated” pixels are discarded and only change pixels that cover a large area are further taken into account.
The next step of the algorithm is to discard pixel differences that are caused by different illumination (e.g. shadows). The corresponding patches are ignored from further processing, while the rest of the (candidate) patches are fed into a deep learning module that can detect four different types of changes. Note that the detectable “types” of changes are defined according to their low-level image content, which only loosely correspond to semantically defined types of changes. The four change detection modules are as follows:
• Blob detection, changes like homogenous blobs are present in only one patch of the pair
• Texture detection, where changes that appear like small-sized changes in the patch texture
• Global detection, which detect global changes between the two patches
• Motion detection, in which a small feature has moved between the two patches
Each module is a supervised learning classifier that produces a “confidence level” score, i.e. a value between 0 and 1 that models the probability of a change of a certain type is present. These intermediate scores are further combined (using a second-layer classifier). The final result for each patch is a score, which is compared with a threshold to declare a change. Finally, the detected changes are expanded so as to cover a 512 x 512 region, so as to provide context for the identified dynamic feature.
The UCL change detection algorithm has been extensively used to detect changes on the imagery that was co-registered and orthorectified within the project, as part of WP4. The change detection results that were acquired using the algorithm of the previous sub-section are as follows:
• 3,365 changes on image pairs from randomly selected image pairs from MC11-E, MC11-W, as well as regions-of-interest all over Mars, a dataset that was also internally released so as to form the input to the WP7 crowdsourcing experiment
• 465 changes from the same datasets that were further manually annotated, generating 270 true positive and 195 false positive results. The implied change detection performance (58.06%) provides a first evaluation of the change detection performance, while a more thorough evaluation will be conducted with the help of planetary scientists and the citizen science inputs. Note that while this score doesn’t appear impressive it still defines the state-of-the-art, since it is the first time that such an algorithm has ever been applied to Mars. Moreover, from discussions with planetary science colleagues, an automatic tool that generates 3 semantically meaningful changes per 5 detected change instances is useful.
• 3,512 changes that were detected by the available high-resolution imagery of the MC11-E half-quadrangle. Note that 527 of them were duplicates with the first set of experiments, therefore they needed to be excluded from the overall number of changes. However, the fact that the algorithm produces duplicates signifies that the results are reproducible, i.e. the stochastic stages of the algorithm don’t cause irreproducibility in the algorithm output, a feature that needs to be carefully designed in any algorithm that employs stochastic stages.
• 4,764 changes that were detected from the available high-resolution imagery of the MC11-W half-quadrangle, 1,089 of which were duplicates with the first set of experiments and needed be excluded from the total change estimation.
Overall, the change detection runs estimated 10,490 in the MC11 area that is 5.5% of the Martian overall surface. Taking into account the preliminary true positive rate of 58.06% and extrapolating to the total size of Mars, this would mean more than 110 thousand instances of change, could be detected in all the imagery acquired by CTX, HRSC, THEMIS-VIS and MOC-NA instruments.
This number confirms the main hypothesis of the project, i.e. that the frequency and distribution of Mars surface changes is so large that any “manual” analysis of image pairs to track dynamic features is unrealistic due to the data-intensive Mars exploration of the last two decades, and that Mars surface science would greatly benefit from the development of automatic change detection techniques that would assist planetary scientists to flag the most “promising” dynamic feature regions. Moreover, it becomes apparent that a thorough examination of 110,000 changes would require a significant amount of time from the planetary science community. Some examples of automatically detected changes are shown in Figure 17 [A new impact crater that was automatically detected in MC11-E half-quadrangle. Images (a) and (c) are THEMIS-VIS while images (b) and (d) are CTX images. The automatic change detection pipeline detected the pairwise change in all pairs (i.e. (a)-(b), (b)-(c) and (c)-(d)), thus successfully tracking the whole process of the impact crater formation. Image (a), which is the last image before the impact, is from 2005 while image (b), which is the first image after the impact, is from 2007. This is a new discovery, not reported in the literature, which becomes more important when considering that the impact happened only 190 km from Opportunity rover. See also Figures 18 a & b [New slope streaks detected over Olympus Mons Aureole].
The change detection pipeline development was shown to be a rather challenging task on its own, given the fact that it required first the successful completion of WP2, WP4 and WP7. The final results of the analysis are going to be initially manually checked at UCL to generate a thorough analysis of the algorithm.
Subsequently, the results will be sent for further assessment to planetary scientists, who will provide a valuable feedback about the results and pinpoint unreported instances of Martian dynamic features, and possibly newly discovered dynamic features. We believe that it is straightforward that the impact that such a dataset can have on the Mars scientific community. This will be one of the main legacies of the iMars project, i.e. that multiple future publications about the Mars surface dynamic processes will originate from the results of the data-mining algorithm that was developed within this project.
In WP6, UOS efforts were concentrated on addressing arguments regarding the migration speeds of Martian dune fields and their correlation with atmospheric circulation. In this aspect, UoS developed a generic procedure [see Figure 19: UoS Implemented algorithms for the observation of dune migration] to precisely measure the migration of dune fields with HIRISE employing a high-accuracy photogrammetric processor and subpixel image correlator. The key features over conventional manual tracking which has been employed by planetary scientists are:
• The introduction of very high resolution orthoimages and stereo analysis based on hierarchical geodetic control for better initial point settings
• Positional error removal throughout sensor model refinement with a non-rigorous bundle block adjustment, which makes possible the co-alignment of all images within a time series
• Improved sub-pixel co-registration algorithms using optical flow with a refinement stage conducted on a pyramidal grid processor and a blunder classifier.
The algorithms were tested on time sequences of high-resolution HIRISE images over a large number of Martian dune fields (see example in Figure 20: Example of the detected migration pattern over the Kaiser dune).
As one of the pilot studies on change detection agreed within the consortium, DLR analysed multi-temporal HiRISE images from the edges of the north polar cap of Mars in order to detect and study changes due to block falls and their implications for processes involving active mass wasting. After a systematic survey of recent block fall events recorded in HiRISE images, this study included an assessment of the geometric quality of the image data products used and the resolution limit of the blocks in images and DTMs. In a next step, detected events were grouped according to their imaging times and conditions, and methods for automatic detection of blocks were tested and applied, with the aim of better defining the sources of the unstable blocks and derive estimates on erosion rates.
WP7 Crowd-sourced features for change discovery and validation of data mining
WP7 concerned the development of Citizen Science activities for using the imagery developed by UCL for WP2, WP3 and WP4 to investigate changes on the Martian surface across the time period during which orbiting spacecraft observations have been undertaken. The intent of WP7 was to provide a means to engage interested members of the general public with these data in a structured manner (relevant to dissemination and iMars WP8 and also to complement the web GIS system developed in WP5), to provide scientific results to support on-going development of algorithmic change detection techniques (WP6) and to extend knowledge about the design of citizen science projects including the provision of guidelines and processes for developing future projects. WP7 introduces a notable interdisciplinary element into the project.
The specific scientific aims were to bring Human Factors knowledge to bear on the novel problem of Martian citizen science, to carry out experiments with humans in support of design decisions about citizen science tasks, to build a citizen science project and develop relevant software as required, validate results using it and to finally report guidelines and results based upon this.
At the start of WP7 UNOTT undertook an extensive review of the literature related to citizen science. This remains a relatively unstudied area in its own right (with some noteworthy exceptions) although significant relevant knowledge exists across a wider multidisciplinary terrain including Human Factors, Computer Science, Cognitive Psychology, Psychophysics, Organisational Theory and Digital Economy. The core reason for this emerges from the recognition that citizen science is the development of an online system of work (albeit work undertaken by volunteers). It involves the optimisation of motivation and performance, the deployment of tasks, designing interfaces that consider human psychological capability and relating all these to the primary scientific goals of the work. In order to bring this diverse literature together, to structure our own design process and to exemplify this novel understanding of the problem space, we devised the model shown below that shows how different facets of citizen science design interrelate and constrain each other and questions of motivation and performance trade-offs should be considered (see Figure 21: A Human Factors system model of Citizen Science).
This model of the functioning of a citizen science system implies a particular form of a development process that emphasises user-centric processes and the representation or advocacy of that user at all points in the process, thus ensuring a balance between the technical and social elements of the resultant sociotechnical system portrayed above. The development process takes a deliberately phased process to ensure that, as the complexity of the system grows, key human aspects remain in focus (see Figure 22: A nine stage citizen science development process used within iMars. Examples of key considerations and tradeoffs are listed alongside each stage with examples from the present work. Note at the third stage that the presence of automation should emphasise rather than detract from the consideration of human issues which become all the more crucial). Again, to our knowledge no standard process or commensurate set of detailed guidance has been identified hitherto in this area. Further discussion of this work, together with our reflections and is available in iMars public deliverable D7.4 and from the specific viewpoint of Human Factors practice in Houghton et al., (2016).
As per the development process below, a specific design issue in this project is how to best design this for change detection. However, change detection has to our knowledge not been a particularly popular form of citizen science task (which tends more often towards annotation, transcription or some form of counting) with little direct precedent. Therefore we carried out a series of experiments to explore key parameters in this activity; how many features a participant should be asked to look for (should they specialise or consider a wide range?); what interface should be used (side-by-side vs. flicker, and how should that flicker be tuned?); should participants know anything about crowd or algorithmic confidence before viewing stimuli or go them afresh; and how comparable is the performance of amateurs and professionals in these tasks? Alongside these quantitative measures we also collected data about perceived workload, trust in the system and qualitative feedback formative to the next iteration. The findings of these experiments are summarised in D7.4.
The final form of the citizen science project was developed iteratively and required in its earlier stages the development of specific functionality to display ‘flickered’ imagery and handle the translation of iMars imagery into a form that could be used within the Zooniverse Panoptes framework. This is accessible in public Github form at: https://github.com/sprinklebum/iMars-eu
The iMars citizen science project (Mars in Motion) can be accessed at: https://www.zooniverse.org/projects/imarsnottingham/mars-in-motion
Key functionality offered by the site, as a result of the development process we followed includes: the ability to manually flicker, automatically flicker and where required by users in the interests of wide accessiblity, compare side-by-side iMars image pairs [see Figure 23: Interface for change detection]:
Social functionality includes the provision for users to comment on features, collect particularly noteworthy or photogenic imagery and to discuss with other users their experiences of participating in the project [see Figure 24: The collection of interesting imagery as a facilitator of participant interaction].
Data from the first 9,000 classifications using Mars In Motion are shown in Table 2, broken down by feature type.
Feature Type N.Detected Mean Volunt. Agr-t Median Volunt. Agr-t Average N.of Volunt.
Dune 105 0.77 0.77 2.07
Dust Devil Track 170 0.84 0.84 2.88
Gully Slide 65 0.53 0.52 2.77
Impact Crater 83 0.63 0.63 2.27
Seasonal Fan 32 0.50 0.50 2.00
Slope Streak / RSL 154 0.75 0.75 2.59
Table 2: Classifications of the first 9,000 change detection images broken down by feature type.
Some examples of changes that have attracted the most interest by volunteers to-date are shown in Figure 25 [Examples of volunteer change detections. From top right clockwise: (a) dust devil tracks, (b) slope streaks, (c) & (d) new impact craters], which include new impacts and dust devil tracks.
WP8 Outreach
WP8 aims at maximising the visibility of the project outcomes with the scientific community and the general public as well as identifying new opportunities which arose as a result of the datasets and the web-GIS and crowd-sourcing mechanisms. WP8 includes three user workshops and the documentation of their implementation and results (tasks T8.1-T8.3). Task T8.4 focuses on the production of animations and videos to present the developments and results of iMars in support of their dissemination. Finally, task T8.5 comprises the set-up and maintenance of the project website by UCL.
A total number of 10 abstracts were submitted to the DPS/EPSC 2016 of which 5 were accepted as oral presentations at the meeting and the remainder as poster presentations. Furthermore, the consortium tried to arrange with the meeting and science organizers to bundle the presentations in one session to highlight their common links to the overall approach of iMars. Unfortunately, this was not granted by the program committee as contributions are accepted and arranged according to individual sessions at DPS conferences with no flexibility to have any special sessions.
Nevertheless, the iMars contributions covered the entire range of topics dealt within the project. Spreading the presentations over different sessions, on the other hand, provided also an advantage, as a wider audience was reached.
The Co-ordinator attended some of the oral presentations. The talk given by Jessica Wardlaw (Sprinks et al., 2016a) was well attended with some 30 people, the talks from Lida Fanara, Jung-Rack Kim and Yu Tao had almost a 100 attendees with lots of follow-up questions raised. The poster session was poorly attended with only 2-3 visitors/poster. Physically it was away from the main part of the poster room and away from the refreshments. Nevertheless, it was a good opportunity to engage with NASA scientists about the iMars results.
The RPIF 3D workshop (T3, WP8) was organized by MSSL/UCL in co-operation with the RPIF UK facility (Director: Prof. Jan-Peter Muller & Data Manager: Dr Petet Grindrod) and Europlanet (Prof. Steve Miller & Dr. Norbert Krupp). Its main objective was to introduce early-career scientists to the range of software tools available to generate 3D data products from planetary exploration data, mainly using data of Mars. In addition to known software libraries and workflows like the NASA-USGS ISIS and Socetset® solution, newly developed software like the modified UCL-NASA Ames Stereo Pipeline, which was developed within the iMars project, was demonstrated. Other topics of this 3 full day workshop included data handling and data fusion, tools for the digitisation of geological and geomorphological features and an introduction to GIS systems. In total 22 international students and young professionals participated in the workshop including 1 participant from Turkey, 4 from Russia, 1 from Spain, 3 from Poland, 1 from Hungary, 1 from Slovenia, 1 from Italy, and 10 from the UK. All these participants volunteered to participate in the beta testing of the Mars in Motion system. All aspects of the iMars project were covered in the presentations, which included:
• Introduction of new datasets from the HRSC experiment
• Co-registration of different NASA data sets to HRSC
• Processing of stereo data into DTMs and data mining
• GIS presentations and tools
• Introduction of Citizen Science platforms for scientific research.
Next to the demonstration of the iMars products, a questionnaire was developed to be answered by the workshop participants. Answers provided guidance on further steps within the project and its dissemination as well as potential spin off projects.
22 participants of the workshop filled in the questionnaire that was handed out at the beginning of this meeting. Expertise and fields of research of the participants was wide spread with a concentration on planetology, geology and GIS [see Figure 26: Distribution of participants’ interests & study fields (left). Right – data participants worked on before the workshop]. All, except two of the participants do work with planetary image data of various sources. However, the majority did have contact with Mars exploration data of some kind. Here it was indicated that the highest resolution orbital imaging data (CTX and HiRISE) are the most used data, followed (as far as Mars data are concerned) by HRSC images from the European Mars Express mission and rover images of NASA’s MSL. The category “Others” includes Martian spectral data as well as orbital data of missions to other planetary bodies like Messenger data or Cassini data.
Based on the questionnaire answers, for most participants of the workshop the focus of their studies appears to be the interpretation of data and data products rather than producing derived products like DTMs themselves [see Figure 27: Fields of application indicated by the workshop participants]. Though 16 out of the 22 participants need to process retrieved data to achieve their tasks.
Data processing in this case includes co-registration, preparatory work to load the data into software, i.e. ArcGIS, radiometric calibration, and also mosaicing to extend the covered area to the area of interest. All this is done for the purposes of final DTM generation, mapping, mosaicing and detailed interpretation of the data. Visual inspection of input data to the processing appears to be a common method to judge the suitability of data for the purpose. Though, some participants do use the meta-information to select their input data.
Finally, 11 short videos were created to promote activities and results of the iMars project beyond the end of the project. These comprised descriptive videos, tutorials, and virtual flights over the reconstructed Martian surface. The videos demonstrated results of the iMars project, and described the developed tools and techniques used to derive the data products. These 11 videos were placed on a dedicated YouTube Channel that can be accessed through https://www.youtube.com/channel/UCnQ_SviuCnJjSQOG8v-Ru1Q as well as through https://arquivo.pt/wayback/20170603225639/http://www.i-mars.eu/outreach/media/.
Potential Impact:
Impact Summary
The data product specifications for HRSC multi-orbit products will form the basis for the systematic processing of multi-orbit HRSC data products by the Mars Express HRSC team. The quadrangle mapping by HRSC multi-orbit data products represents a major effort in the cartography of Mars, performed by the Mars Express HRSC Team using the complete mission data record, and will be available to the science community and general public through the ESA Planetary Science Archive (PSA) and NASA Planetary Data System (PDS). The results of the investigations performed within iMars on quality-related aspects and validation have been directly communicated to the HRSC Team.
UoS efforts and subsequent research work based on the outcomes and softwares of the WPs contributed to the planetary science community included (a) demonstrating the importance of precisely controlled topographic products over a planetary surface for quantitative analyses as well as for simulation, measurement and modelling; (b) how to validate the conventional tools in planetary surface interpretation with other approaches. For instance, UoS proposed the validation of previous results of martian dune migrations based on a new method. It identified many highly important missing points, which was caused by the absence of the technological understanding by the researchers.
Technical foregrounds from UoS, especially WP1 and WP6 are already being applied to addressing certain terrestrial environmental issues. The software routines developed by UoS efforts for WP1 stereo processor implementation have been applied for a Greenland mapping project. A number of spaceborne optical stereo and radargrammetric DTMs over southern Greenland were created using a variant software based on WP1 UoS stereo line and used for the scientific research works. WP6 software by UoS was tested over the Kubuchi desert in northwestern China where dust storms frequently occur and cause significant problems in public health over all of northeastern Asia. The initial test was very successful and the further scientific utilization of WP6 UoS software has gained the support of an international organization.
For the real world applications of foreground by UoS, especially HD video contents and their interfaces, UoS has contacted potential industrial partners and public bodies as the possible consumers. Although it still requires more investments to achieve realistic outcomes, such efforts clearly arouse more public/industrial attention for the international Martian studies and their applications.
The CASP-GO, ACRO, and change detection software developed in WPs 1 & 4 and WPs 2 & 6 are a valuable new resource to the scientific community of geologists who are interested in the exploitation of DTMs and co-registered image sequences showing changes over the last 40 years. They have huge potential for EO imagery as well as application to other planetary bodies such as Mars and the Moon. CASP-GO will be released on GitHub for maintenance by NASA Ames later in 2017. The NASA data products from WP4 will be converted into PDS4-complaint format and released through the NASA PDS Imaging node at JPL whilst it is planned that the new data products from HRSC for the SPRC will be released through the ESA PSA. The webGIS will be the principal means whereby members of the scientific community can visualise and access iMars derived products until they have been integrated into the PDS and PSA systems. Some of the iMars products are still in production as they do not require very much manual supervision or intervention, such as completing the DTM production of all the CTX stereo-pairs with 100% overlap, all the HiRISE stereo-pairs which have ≥5 repeat images over the same location and if some mechanism can be found for student interns completing the production of the overlapping NASA images over some 45% of the planet which HRSC has mapped to date. Change detection is likely to require new resources in the future to be applied to the remaining 45% of the planetary surface. Initially, it is expected that most science exploitation will take place through personal collaborations but it is anticipated that over time as the community learns of the value of the resource that greater and greater use will take place independent of the iMars consortium, especially once the data are integrated into the PDS & PSA infrastructure.
The findings of the work into the design of citizen science have wider application in the area of ‘human computation’. In contrast to citizen science, human computation describes the paid execution of small decomposed ‘task units’ by humans using online platforms such as the Amazon Mechanical Turk and Crowdflower. This might include activities such as annotating documents, looking up information on the web, translating short phrases and detecting features in imagery (similar tasks are obviously undertaken under the rubric of citizen science albeit for different reasons). The design of these platforms is a topic of some controversy and debate, particularly owing to concerns about the economic fairness of payment schemes and the experience of workers themselves, particularly as the extremes of task decomposition employed, leave workers doing fragmentary, momentary and generally unrewarding work without much autonomy, community or a sense of the wider impact of their labour. As Kittur et al., (2013) ask: “Can we foresee a future crowd workplace in which we would want our children to participate?” The root cause of these problems is that the work is not as such designed for humans, as designed around what cannot be otherwise automated (this is known as ‘left-over task allocation’). This is arguably not dissimilar from a digital revisiting of Taylorism and Fordism, historically forms of work design associated with worker demotivation, drudgery, strained industrial relations and flawed attempts to reconcile human activity with that of a factory machine (see Sharples & Houghton, 2017 for a discussion). We are reminded of the latter forcibly by accounts of online work that refer to people in dehumanising terms such as ‘human processing units’, conceptualising them not as machinery now but as elements of computer architecture. Recast from this perspective, work in the related area of volunteer citizen science allowed us to examine some of these issues in online task execution and its relationship with automation in a far more benign context where the onus was on us to please volunteers rather than for workers to please employers. If, as many predict, human computation expands as a form of work as a concomitant to increased automation in the workplace, the humanistic and user-centred ways of thinking about online work and the development processes developed and demonstrated within this project will be useful correctives emphasising the role of the human and providing a design process towards a human-automation collaboration that delivers good task performance for the mutual benefit of workers, businesses and citizens.
Dissemination activities carried out during the project were coordinated in a dedicated WP, which included contributions from all project partners. Dissemination activities were aiming at increasing the visibility of project aims and results to the science community and general public, at identifying new opportunities which arose as a result of the datasets and the web-GIS and crowd-sourcing mechanisms, and at implementing efficient communication with the scientific community and general public for enhancing and promoting the use and benefit of iMars tools and data. An additional aim was to employ the NASA Regional Planetary Imaging Facilities (RPIFs) to promote the products and tools developed by iMars. This required continuing efforts over the entire lifetime of the project and was achieved by implementing (a) a series of user workshops and a dedicated science session at a major planetary science conference (EPSC 2015) (b) a series of videos accessible through a public on-line channel (c) the iMars website https://arquivo.pt/wayback/20170603204309/http://www.i-mars.eu/
The prospects for future application of iMars topics and tools show high potential according to feedback gained within the project through several dissemination events and exchange of experience ideas with potential users through presentations, questionnaires, and workshops. Since the developments accomplished in the project have been flanked by several dedicated user workshops, including evaluation of user feedback returned, an enhanced compliance with user requirements and user interests can be expected. The high interest in the community for the topic of surface changes was confirmed throughout the numerous presentations of the project at workshop and technical sessions of planetary science conferences. The plan for a wide and unrestricted access to data products through the PSA and PDS archives will further support future studies in the area. The project has also achieved a high impact on public awareness for current space science and its methodology through the realisation of a citizen science study and its promotion via the different dissemination events.
Three user workshops where held during the project:
• User Requirements Workshop, at the European Geophysical Union (EGU) General Assembly, May 2014, Vienna (Austria)
• First User Consultation workshop, at the European Planetary Science Conference (EPSC), September 2015, Nantes (France)
• Final User Consultation workshop, at UCL (Great Britain), June 2016, in conjunction with the RPIF 3D workshop
The objective of the first user workshop (User Requirements Workshop) was to introduce the project’s aims and developments to the scientific community and in particular potential users, and assess user requirements and interests. The specific objective of the second workshop (First User Consultation workshop) consisted in presenting already accomplished project developments and obtaining user feed-back to guide the further project developments. Finally, the objective of the third workshop consisted in presenting the project results to users, gain feed-back to guide final implementation and analysis activities as well as evaluate exploitation opportunities. The workshop included presentations on the contents and aims of iMars, initiation of a user survey developed by he consortium, and open discussion with potential users recorded by taking minutes. The survey was accessible in a printable version as well as an on-line version hosted on the iMars website. The workshop minutes and the survey responses were analysed and documented in a report, which was made publicly accessible on the iMars website. The conclusions concerning user requirements were considered as useful guidelines for the technical development tasks of iMars. Experience was gained concerning the expected return of user surveys involving technical aspects of particular relevance, in terms of number of responses. The high interest in the community for the topic of surface changes was confirmed and the workshop and survey helped to define a set of specific case studies in this field and obtain feedback form researchers actively engaged in the area.
The second workshop was organized in a special technical conference session and a demonstration event at EPSC 2015. It was organized jointly with FP7 projects CROSS-DRIVE and PRoVIDE to highlight the complementarity of approaches and maximize the outreach to a potentially large user group. The direct feedback gained from discussions during the technical session and the demonstration session were documented in a publicly accessible workshop report together with an evaluation of “lessons learned” and updated information on user requirements. The technical session was part of the official conference technical program (MT8, Zooming in-and-out of Mars: new tools to interact with multi-resolution Mars datasets, conveners K. Willner, J.-P. Muller, S. Gupta) and included 11 oral presentations and 8 posters in two scientific sessions on Sept. 29 and 30, 2015.
The third workshop consisted of a hands-on workshop organised in collaboration with the UK NASA RPIF at Mullard Space Science Laboratory (MSSL) in June 2016. It was held in conjunction with the EUROPLANETS 2020 NA1 networking activity hosted at UCL-MSSL and led by the Director of the UK NASA RPIF, Prof. Muller on 3D Imaging. All of the 2nd day and part of the 3rd day (8-9 June 2016) were devoted to showing-off the iMars system, including CASP-GO, ACRO, the webGIS, data mining and getting users feedback to a questionnaire developed by the consortium. The 3rd day of the workshop included capturing the scene classifier data required for WP6 using the prototype “Mars in motion” website. The audience of the 3rd workshop consisted mainly of early-career scientists from different European and non-European countries (1 from Turkey, 4 from Russia, 5 from Inclusiveness countries, 1 from Spain, 10 UK).
The final outreach event organised in London, took place on March 14, 2017, 14 presentations on iMars tools and applications, presented by iMars partners and external colleagues to an audience of 70 interested members of the public and colleagues from the planetary science community.
A specific effort within WP8 was devoted to the production of a collection of 11 videos accessible on a public on-line channel. The topics addressed by the videos include the description of tools developed within iMars, the explanation of techniques applied, and the presentation of data products. Examples include the demonstration of the new HRSC colour products in 3D for 3 different areas, a video showing how mosaics are produced and a flight over the new MC-11 mosaics, a video demonstrating 3D interactive visualisation tools being used to show off the new co-registered HRSC, CTX and HiRISE 3D products and the EU-FP7 PRoViDE interactive web-based tool, Pro3D® developed for 3D digitisation of HiRISE 3D products. In addition, videos to show the CASP-GO, ACRO and data mining tools were developed as well as for the QGIS extensions to allow the visualisation of SHARAD and MARSIS subsurface radar echograms. Finally videos were produced to show the citizen science “Mars in Motion” tools.
List of Websites:
https://arquivo.pt/wayback/20170603204309/http://www.i-mars.eu/
iMars Co-ordinator: Professor Jan-Peter Muller, j.muller@ucl.ac.uk