Periodic Reporting for period 2 - NOSTROMO (Next-Generation Open-Source Tools for ATM Performance Modelling and Optimisation)
Période du rapport: 2021-06-01 au 2022-11-30
The Air Traffic Management (ATM) system is composed of a myriad of elements that interact with each other, including interdependent policies and regulations, stakeholders, technologies and market conditions. These interactions give rise to a number of properties characteristic of complex adaptive systems, such as non-linearity, emergence and adaptation, which make the ATM system intrinsically difficult to model one of the most challenging modelling problems: the assessment of the performance impact of new solutions at a system-wide level.
The development of methodologies to evaluate the impact of new ATM concepts and technologies on high-level, system wide Key Performance Indicators (KPIs) has been a long-time objective of the ATM research community. Low-level validation activities based on fast-time simulation, human-in-the-loop (HITL) simulation, shadow-mode trials and live trials provide accurate estimates of the performance of a certain solution in a given operational environment; however, implementing such validation approaches for different combinations of solutions at a network-wide scale is infeasible, or at least prohibitive in terms of both cost and time. It is therefore necessary to resort to performance models that consolidate the results of low-level validation experiments conducted for different solutions at a local level and estimate the integrated impact of such solutions at network level.
In this context, the goal of NOSTROMO is to develop, demonstrate and evaluate an innovative modelling approach for the rigorous and comprehensive assessment of the performance impact of future ATM concepts and solutions at ECAC network level. This approach will bring together the ability of bottom-up microscopic models to capture emergent behaviour and interdependencies between different solutions with the level of tractability and interpretability required to effectively support decision-making. The specific objectives of the project are the following:
1. Develop a methodology for the construction of ATM performance metamodels that approximate the behaviour of computationally expensive simulation models so as to allow a systematic and efficient exploration of the model input-output space and a robust handling of the uncertainty associated with the model predictions, by exploiting recent advances in the field of active learning.
2. Implement and validate the proposed metamodeling methodology by developing metamodels of two state-of-the-art microsimulation tools (namely Mercury and FLITAN) able to reproduce ATM performance at ECAC level.
3. Develop a set of visualisation and visual analytics tools that facilitate the analysis, interpretation and communication of the results of the new performance metamodels.
4. Demonstrate and evaluate the maturity of the NOSTROMO approach and the capabilities of the newly developed toolset through a set of case studies addressing the performance assessment of SESAR Solutions at ECAC level. The case studies shall cover a variety of ATM phases, solutions and KPAs/KPIs sufficiently heterogeneous to allow a comprehensive benchmarking against the performance modelling methodologies currently in use, with the aim to analyse the added value and the limitations of the NOSTROMO approach and evaluate the appropriateness of its transition to SESAR IR and its potential to improve of the European Operational Concept Validation Methodology (E-OCVM).
NOSTROMO project has been developed in an incremental approach towards the objectives, by evaluating and refining the proposed methodology in an iterative manner in the light of the results obtained in its specific applications.
The development of methodologies to evaluate the impact of new ATM concepts and technologies on high-level, system wide Key Performance Indicators (KPIs) has been a long-time objective of the ATM research community. Low-level validation activities based on fast-time simulation, human-in-the-loop (HITL) simulation, shadow-mode trials and live trials provide accurate estimates of the performance of a certain solution in a given operational environment; however, implementing such validation approaches for different combinations of solutions at a network-wide scale is infeasible, or at least prohibitive in terms of both cost and time. It is therefore necessary to resort to performance models that consolidate the results of low-level validation experiments conducted for different solutions at a local level and estimate the integrated impact of such solutions at network level.
In this context, the goal of NOSTROMO is to develop, demonstrate and evaluate an innovative modelling approach for the rigorous and comprehensive assessment of the performance impact of future ATM concepts and solutions at ECAC network level. This approach will bring together the ability of bottom-up microscopic models to capture emergent behaviour and interdependencies between different solutions with the level of tractability and interpretability required to effectively support decision-making. The specific objectives of the project are the following:
1. Develop a methodology for the construction of ATM performance metamodels that approximate the behaviour of computationally expensive simulation models so as to allow a systematic and efficient exploration of the model input-output space and a robust handling of the uncertainty associated with the model predictions, by exploiting recent advances in the field of active learning.
2. Implement and validate the proposed metamodeling methodology by developing metamodels of two state-of-the-art microsimulation tools (namely Mercury and FLITAN) able to reproduce ATM performance at ECAC level.
3. Develop a set of visualisation and visual analytics tools that facilitate the analysis, interpretation and communication of the results of the new performance metamodels.
4. Demonstrate and evaluate the maturity of the NOSTROMO approach and the capabilities of the newly developed toolset through a set of case studies addressing the performance assessment of SESAR Solutions at ECAC level. The case studies shall cover a variety of ATM phases, solutions and KPAs/KPIs sufficiently heterogeneous to allow a comprehensive benchmarking against the performance modelling methodologies currently in use, with the aim to analyse the added value and the limitations of the NOSTROMO approach and evaluate the appropriateness of its transition to SESAR IR and its potential to improve of the European Operational Concept Validation Methodology (E-OCVM).
NOSTROMO project has been developed in an incremental approach towards the objectives, by evaluating and refining the proposed methodology in an iterative manner in the light of the results obtained in its specific applications.
The project was organised around five case studies and followed an incremental approach based on three iterations. The results obtained during the development and execution of the different case studies served to refine the proposed methodology in an iterative manner.
As main achievements of the project, the following items can be highlighted:
• Development of an active learning metamodeling methodology in the context of the SESAR Performance Framework aiming at reducing the computational burden often associated with fast-time simulation-based studies. This methodology is not expected to substitute the traditional simulation tools but instead to complement the current state of practice of ATM performance assessment.
• Development of a prototypal API enabling the employment of the proposed methodology and the integration of ATM simulators, which may support a future common SESAR integrated simulation and metamodelling platform for performance assessment, visualisation, and decision support.
• Proposal of some guidelines for future simulation models to be developed in the scope of SESAR, ensuring, to the maximum extent possible, compatibility with the NOSTROMO architecture.
• Development of an interactive dashboard that facilitates the understanding, analysis, and communication of the ATM performance metamodel results.
• Cost benefit analysis of the application of the metamodeling methodology proposed by NOSTROMO.
The final results of the project with real case studies and the most complex Solutions selected during the project showed that the metamodeling approach followed by NOSTROMO provides results very close to the simulator with much less computational time. The NOSTROMO metamodels allow a deeper assessment of a solution, amplifying the exploration of the simulation input and output behaviour space and helping to identify patterns and trends.
As main achievements of the project, the following items can be highlighted:
• Development of an active learning metamodeling methodology in the context of the SESAR Performance Framework aiming at reducing the computational burden often associated with fast-time simulation-based studies. This methodology is not expected to substitute the traditional simulation tools but instead to complement the current state of practice of ATM performance assessment.
• Development of a prototypal API enabling the employment of the proposed methodology and the integration of ATM simulators, which may support a future common SESAR integrated simulation and metamodelling platform for performance assessment, visualisation, and decision support.
• Proposal of some guidelines for future simulation models to be developed in the scope of SESAR, ensuring, to the maximum extent possible, compatibility with the NOSTROMO architecture.
• Development of an interactive dashboard that facilitates the understanding, analysis, and communication of the ATM performance metamodel results.
• Cost benefit analysis of the application of the metamodeling methodology proposed by NOSTROMO.
The final results of the project with real case studies and the most complex Solutions selected during the project showed that the metamodeling approach followed by NOSTROMO provides results very close to the simulator with much less computational time. The NOSTROMO metamodels allow a deeper assessment of a solution, amplifying the exploration of the simulation input and output behaviour space and helping to identify patterns and trends.
The results and conclusions of the application of metamodelling approach proposed by NOSTROMO and the evaluation activities were used to develop a consolidated set of guidelines and applicability methodology for the integration of the NOSTROMO models and tools into the SESAR3 IR programme.
To maximise its transferability to SESAR3 IR programme, NOSTROMO conducted a joint research effort with SESAR ER SIMBAD project. SIMBAD has also addressed the use of machine learning techniques to improve ATM simulation and performance assessment, working with EUROCONTROL’s R-NEST simulation tool, which will be one of the main simulation tools for performance assessment in SESAR3. In this joint research effort, NOSTROMO’s metamodelling methodology was used to develop a metamodel of R-NEST, demonstrating the potential benefits of this approach in SESAR3 performance assessment activities. At the moment of the development of this deliverable, the work performed in NOSTROMO and SIMBAD projects has been taken as candidate to be taken up in the SESAR3 activities related to the Master Plan and Performance Assessment, through two SESAR 3 proposals for the IR1 Call, AMPLE3 and PEARL respectively. In particular, the NOSTROMO approach has been proposed to be involved in the development of Optimised Deployment Scenarios in AMPLE3. Additionally, the performance dashboard to be developed in PEARL will build on the visualisation tools developed by NOSTROMO, among other relevant inputs.
For the transition of the NOSTROMO Methodology to the SESAR3 IR Programme a number of improvements will be required, such as the use of more active learning strategies, the development of APIs for other ATM simulators and the integration of multiple simulators in a single metamodel.
To maximise its transferability to SESAR3 IR programme, NOSTROMO conducted a joint research effort with SESAR ER SIMBAD project. SIMBAD has also addressed the use of machine learning techniques to improve ATM simulation and performance assessment, working with EUROCONTROL’s R-NEST simulation tool, which will be one of the main simulation tools for performance assessment in SESAR3. In this joint research effort, NOSTROMO’s metamodelling methodology was used to develop a metamodel of R-NEST, demonstrating the potential benefits of this approach in SESAR3 performance assessment activities. At the moment of the development of this deliverable, the work performed in NOSTROMO and SIMBAD projects has been taken as candidate to be taken up in the SESAR3 activities related to the Master Plan and Performance Assessment, through two SESAR 3 proposals for the IR1 Call, AMPLE3 and PEARL respectively. In particular, the NOSTROMO approach has been proposed to be involved in the development of Optimised Deployment Scenarios in AMPLE3. Additionally, the performance dashboard to be developed in PEARL will build on the visualisation tools developed by NOSTROMO, among other relevant inputs.
For the transition of the NOSTROMO Methodology to the SESAR3 IR Programme a number of improvements will be required, such as the use of more active learning strategies, the development of APIs for other ATM simulators and the integration of multiple simulators in a single metamodel.