Periodic Reporting for period 2 - VESTEC (Visual Exploration and Sampling Toolkit for Extreme Computing)
Berichtszeitraum: 2020-03-01 bis 2022-02-28
Natural disasters such as wildfires all over the world are increasing. At the same time, diseases such as the current COVID-19 have shown the potential impact of widespread transmission of human infectious diseases on modern societies. Such dramatic events extremely threaten human lives and the social economy. To reduce the impacts, governments and aid organizations need reliable information to decide for efficient counter-measures.
Due to continuous advances in high-performance-computing, enhanced opportunities to model and simulate such physical, social, or economic phenomena are available. The produced predictions can help to create control systems for the management of such critical situations. By enhancing these simulations with sensor data, ever more precise and reliable predictions can be generated. Additionally, multiple simulations can be gathered into ensembles to increase statistical validity. Nevertheless, the integration of such complex and interactive workflows for urgent decision making, exploiting national or international HPC resources, is poor. The main reason is that such systems are not designed for interactive and urgent computing scenarios.
The overall objective of VESTEC is to bring such computational models, data fusion concepts, and ensemble analysis into workflows for urgent decision making that can save lives and reduce economic loss. The main challenge is to support and enable integrated, interactive access to todays and future extreme computing environments. Therefore, VESTEC integrates novel approaches to heterogeneous data fusion, urgent computing, and suitable data analysis methods together with visualizations to present the right information in near real-time to decision makers.
Due to continuous advances in high-performance-computing, enhanced opportunities to model and simulate such physical, social, or economic phenomena are available. The produced predictions can help to create control systems for the management of such critical situations. By enhancing these simulations with sensor data, ever more precise and reliable predictions can be generated. Additionally, multiple simulations can be gathered into ensembles to increase statistical validity. Nevertheless, the integration of such complex and interactive workflows for urgent decision making, exploiting national or international HPC resources, is poor. The main reason is that such systems are not designed for interactive and urgent computing scenarios.
The overall objective of VESTEC is to bring such computational models, data fusion concepts, and ensemble analysis into workflows for urgent decision making that can save lives and reduce economic loss. The main challenge is to support and enable integrated, interactive access to todays and future extreme computing environments. Therefore, VESTEC integrates novel approaches to heterogeneous data fusion, urgent computing, and suitable data analysis methods together with visualizations to present the right information in near real-time to decision makers.
The project aimed at developing and integrating urgent decision-making scenarios as an emerging use mode for high-performance computing. In WP2, the VESTEC system has been designed as a fundamental basis to integrate the current and future use cases. One major challenge when dealing with extreme data is efficient analysis. Especially for urgent scenarios, it is necessary to provide the right information at the right time. Therefore, WP3 has developed and integrated in-situ data analysis methods to reduce data sizes and highlight prominent features. A novel and progressive computation method for generating topological proxies has been investigated. On top of the generated proxies, a reduction approach based on computing barycenters between persistence diagrams has been developed. The integrated feature-based statistical methods provide the basis for analysis of simulation ensembles.
Another important functionality for urgent decision-making is visualization. This post-processing presents results to the decision makers and supports to find the right decisions for efficient counter-measures. Therefore, WP4 provided interactive and explorative visualization tools and algorithms. The classical scientific data visualization has been extended by statistical data presentation based on persistence diagrams. These diagrams can be filtered interactively to extract important clusters of information. These are called topological proxies and presented as georeferenced graphical objects in the visualization for the decision makers. The meaning is different for the different use cases. The usefulness has been discussed with the domain experts.
Current HPC systems are not designed for urgent computing. This limitation mainly originates trough the usage of classical batch systems. Additionally, HPC systems are isolated and processing external data or executing co-simulations becomes challenging. The goal of WP5 has been to support timely and complex urgent computing workflows to overcome these limitations. Therefore, a federator approach was implemented. This federator is able to manage multiple national or international HPC system which helps to reduces considerably the waiting time before those jobs are schedule for execution. This included also the incorporation of interfaces for the processing of external data.
WP6 integrated the methods from previous work packages to establish and execute all pilot applications. Furthermore, simulation codes have been extended and optimized to deal with the in-situ co-processing. With that, important results for the decision makers can now be processed immediately on the supercomputing and stream to the crises centers for interactive analysis. For all our three use-case, we have developed story lines to guide the goal-driven development and execution. The requirements for the features to be extracted were pretty different but made use of the same workflow offered by the VESTEC system, which shows the generic approach and the capabilities to be extended for further crisis scenarios. In dedicated execution evaluation events, the use cases could proof the efficiency and use-case specify outcome for urgent decision support.
Another important aspect of VESTEC is to make outcomes available to the public resources and build communities interested in these topics. WP7 aimed to reach these goals. The main achievements have been the implementation of the project website, establishing a Twitter channel, organization of workshops at international conferences. Besides these activities many scientific publications have been created and made available as open access whenever possible. Additionally, tutorials and hackathons were held. Eventually, we could present all our findings and implementations on our final workshop, which had to be carried out as a virtual event because of the current pandemic situation. A concluding journal paper is currently in preparation which will present the contribution and produced result by all partner of the consortium.
Another important functionality for urgent decision-making is visualization. This post-processing presents results to the decision makers and supports to find the right decisions for efficient counter-measures. Therefore, WP4 provided interactive and explorative visualization tools and algorithms. The classical scientific data visualization has been extended by statistical data presentation based on persistence diagrams. These diagrams can be filtered interactively to extract important clusters of information. These are called topological proxies and presented as georeferenced graphical objects in the visualization for the decision makers. The meaning is different for the different use cases. The usefulness has been discussed with the domain experts.
Current HPC systems are not designed for urgent computing. This limitation mainly originates trough the usage of classical batch systems. Additionally, HPC systems are isolated and processing external data or executing co-simulations becomes challenging. The goal of WP5 has been to support timely and complex urgent computing workflows to overcome these limitations. Therefore, a federator approach was implemented. This federator is able to manage multiple national or international HPC system which helps to reduces considerably the waiting time before those jobs are schedule for execution. This included also the incorporation of interfaces for the processing of external data.
WP6 integrated the methods from previous work packages to establish and execute all pilot applications. Furthermore, simulation codes have been extended and optimized to deal with the in-situ co-processing. With that, important results for the decision makers can now be processed immediately on the supercomputing and stream to the crises centers for interactive analysis. For all our three use-case, we have developed story lines to guide the goal-driven development and execution. The requirements for the features to be extracted were pretty different but made use of the same workflow offered by the VESTEC system, which shows the generic approach and the capabilities to be extended for further crisis scenarios. In dedicated execution evaluation events, the use cases could proof the efficiency and use-case specify outcome for urgent decision support.
Another important aspect of VESTEC is to make outcomes available to the public resources and build communities interested in these topics. WP7 aimed to reach these goals. The main achievements have been the implementation of the project website, establishing a Twitter channel, organization of workshops at international conferences. Besides these activities many scientific publications have been created and made available as open access whenever possible. Additionally, tutorials and hackathons were held. Eventually, we could present all our findings and implementations on our final workshop, which had to be carried out as a virtual event because of the current pandemic situation. A concluding journal paper is currently in preparation which will present the contribution and produced result by all partner of the consortium.
The VESTEC federator approach established on an upper application layer that successfully increased the interactivity for urgent computing scenarios. A novel algorithm for progressive computation of persistence diagrams has impact to many other application domains dealing with scalar data. Additionally, the novel methods improve the analysis of simulation ensembles also used in varying application domains.
Since natural disasters, such as wildfires and global diseases, are increasing, modelling and simulating such phenomena becomes more and more important to save human lives and prevent socio-economic loss. To advance disaster modelling, VESTEC published a novel fire spread model considering uncertainties in wildfire simulations. Additionally, a mathematical model to simulate mosquito-borne diseases such as Ebola has been developed, which might be adoptable to other kind of diseases. Finally, the space weather code has been enhanced by the in-situ interface which helps to process huge simulation result and stream core features to the end users.
The urgent supercomputing workshops and the final VESTEC workshop have shown that the topics of VESTEC is of increase interests in the urgent computing community. We believe that other applications domains in these fields will benefit from the developments as well. Also, standard bodies and other relevant research programs and frameworks will profit. The suppression activities will be safer and both, strategies and tactics of crisis forces will be more advanced in time and space. The use case owners could incorporate the results in their tools and processes which guarantees the long-lasting application in really crises scenarios.
Since natural disasters, such as wildfires and global diseases, are increasing, modelling and simulating such phenomena becomes more and more important to save human lives and prevent socio-economic loss. To advance disaster modelling, VESTEC published a novel fire spread model considering uncertainties in wildfire simulations. Additionally, a mathematical model to simulate mosquito-borne diseases such as Ebola has been developed, which might be adoptable to other kind of diseases. Finally, the space weather code has been enhanced by the in-situ interface which helps to process huge simulation result and stream core features to the end users.
The urgent supercomputing workshops and the final VESTEC workshop have shown that the topics of VESTEC is of increase interests in the urgent computing community. We believe that other applications domains in these fields will benefit from the developments as well. Also, standard bodies and other relevant research programs and frameworks will profit. The suppression activities will be safer and both, strategies and tactics of crisis forces will be more advanced in time and space. The use case owners could incorporate the results in their tools and processes which guarantees the long-lasting application in really crises scenarios.