Periodic Reporting for period 2 - SIMBAD (Combining Simulation Models and Big Data Analytics for ATM Performance Analysis)
Reporting period: 2022-01-01 to 2022-12-31
The goal of SIMBAD was to develop and evaluate a set of machine learning approaches aimed at providing state‑of-the-art ATM microsimulation models with the level of reliability, tractability and interpretability required to effectively support performance evaluation at network level. The project focused on three fundamental problems:
- how to estimate hidden variables (e.g. variables related to airspace user preferences) that are not directly observable;
- how to select a sufficiently representative set of traffic scenarios so as to take into account the different traffic patterns observed along the year without having to simulate every single day;
- how to build performance metamodels that are conceptually simpler and less computationally costly than microsimulation models, in order to enable a more efficient exploration of the simulation space and an easier interpretation of the modelling outcomes.
The specific objectives of the project were the following:
1. Explore the use of machine learning techniques for the modelling of trajectories and the estimation of hidden variables from historical air traffic data.
2. Develop new machine learning algorithms for traffic pattern classification.
3. Investigate the use of active learning metamodelling to enable a more efficient exploration of the input‑output space of complex ATM simulation models.
4. Demonstrate and evaluate the newly developed techniques in order to assess their maturity, derive recommendations on how to apply them to ATM performance assessment, and propose a roadmap for the transition of the project results to the next stages of the R&D cycle.
- how to estimate hidden variables (e.g. variables related to airspace user preferences) that are not directly observable;
- how to select a sufficiently representative set of traffic scenarios so as to take into account the different traffic patterns observed along the year without having to simulate every single day;
- how to build performance metamodels that are conceptually simpler and less computationally costly than microsimulation models, in order to enable a more efficient exploration of the simulation space and an easier interpretation of the modelling outcomes.
The specific objectives of the project were the following:
1. Explore the use of machine learning techniques for the modelling of trajectories and the estimation of hidden variables from historical air traffic data.
2. Develop new machine learning algorithms for traffic pattern classification.
3. Investigate the use of active learning metamodelling to enable a more efficient exploration of the input‑output space of complex ATM simulation models.
4. Demonstrate and evaluate the newly developed techniques in order to assess their maturity, derive recommendations on how to apply them to ATM performance assessment, and propose a roadmap for the transition of the project results to the next stages of the R&D cycle.
SIMBAD has delivered an ATM performance modelling framework aimed at enhancing the capabilities of large-scale ATM microsimulation models to effectively support performance evaluation at network level. The framework enables the comprehensive exploration and understanding of the performance impact of ATM concepts and solutions under different traffic conditions. The framework comprises three main elements:
1) A model for the estimation of hidden variables related to airspace user behaviour that are necessary inputs for the ATM microsimulation models, such as cost index and landing weight. Different machine learning algorithms for the estimation of hidden variables were implemented, assessing how the estimations of hidden variables provided by the machine learning models can support the mechanistic prediction of trajectory-related KPIs. The validation exercises conducted by SIMBAD show how a more accurate estimation of these hidden variables improves trajectory simulation by providing a more realistic representation of airspace users' behaviour and preferences.
2) A methodology to identify representative traffic patterns at different scales for each particular problem under study. This methodology was demonstrated and validated by applying it to two SESAR solutions: Free-Routing and Demand and Capacity Balancing. For each solution, representative traffic patterns were identified at ECAC, ANSP, ACC, and airport level. Then, for each pattern found, the most representative day of the pattern and the day with the highest deviation were identified. The validation exercises proved that the SIMBAD approach leads to a much more accurate estimation of certain KPIs (e.g. annual fuel consumed in the ECAC area) that than enabled by a selection of representative traffic days based on expert judgement, while reducing the number of required simulation scenarios.
3) A metamodelling framework that enables the approximation of the results of a microsimulation model to facilitate a more efficient exploration of its input-output space. A simulation metamodel of UPC's DYNAMO trajectory simulation tool and two simulation metamodels of EUROCONTROL's R-NEST simulation tool were defined and implemented. The metamodels were developed in collaboration with the NOSTROMO project, taking advantage of the synergies between the metamodelling activities of both SESAR projects. The predictive performance of the trained metamodels was assessed and analysed, showing their ability to accurately reproduce the behaviour of the microsimulation models while drastically reducing the required computational resources.
The SIMBAD Performance Modelling Framework were demonstrated through a series of case studies in which the different components of the framework (either alone or combined with each other) have been put at work. These demonstration exercises showed how the project results can help overcome some of the main drawbacks of traditional
ATM microsimulation models, paving the way for the SIMBAD performance modelling framework to reach higher TRLs and be effectively used in the context of SESAR Industrial Research.
1) A model for the estimation of hidden variables related to airspace user behaviour that are necessary inputs for the ATM microsimulation models, such as cost index and landing weight. Different machine learning algorithms for the estimation of hidden variables were implemented, assessing how the estimations of hidden variables provided by the machine learning models can support the mechanistic prediction of trajectory-related KPIs. The validation exercises conducted by SIMBAD show how a more accurate estimation of these hidden variables improves trajectory simulation by providing a more realistic representation of airspace users' behaviour and preferences.
2) A methodology to identify representative traffic patterns at different scales for each particular problem under study. This methodology was demonstrated and validated by applying it to two SESAR solutions: Free-Routing and Demand and Capacity Balancing. For each solution, representative traffic patterns were identified at ECAC, ANSP, ACC, and airport level. Then, for each pattern found, the most representative day of the pattern and the day with the highest deviation were identified. The validation exercises proved that the SIMBAD approach leads to a much more accurate estimation of certain KPIs (e.g. annual fuel consumed in the ECAC area) that than enabled by a selection of representative traffic days based on expert judgement, while reducing the number of required simulation scenarios.
3) A metamodelling framework that enables the approximation of the results of a microsimulation model to facilitate a more efficient exploration of its input-output space. A simulation metamodel of UPC's DYNAMO trajectory simulation tool and two simulation metamodels of EUROCONTROL's R-NEST simulation tool were defined and implemented. The metamodels were developed in collaboration with the NOSTROMO project, taking advantage of the synergies between the metamodelling activities of both SESAR projects. The predictive performance of the trained metamodels was assessed and analysed, showing their ability to accurately reproduce the behaviour of the microsimulation models while drastically reducing the required computational resources.
The SIMBAD Performance Modelling Framework were demonstrated through a series of case studies in which the different components of the framework (either alone or combined with each other) have been put at work. These demonstration exercises showed how the project results can help overcome some of the main drawbacks of traditional
ATM microsimulation models, paving the way for the SIMBAD performance modelling framework to reach higher TRLs and be effectively used in the context of SESAR Industrial Research.
SIMBAD advanced the state-of-the-art in three main areas:
- Development of new techniques for better estimation of hidden variables, which provide new knowledge on airspace user preferences and constraints, helping understand the actual performance impact of new ATM concepts or solutions on airspace users.
- Development of new techniques for the comprehensive multi-scale classification of traffic patterns in the ECAC area, which enables an enhanced understanding of the performance impact of a certain operational concept under different traffic conditions, thus facilitating more accurate and comprehensive cost-benefit analyses.
- Introduction to active learning metamodelling techniques in ATM, which can be used to translate a complex ATM simulation model into a performance metamodel, improving computational tractability and interpretability of results by allowing a more efficient exploration of the simulator’s behaviour.
The SIMBAD Performance Modelling Framework facilitates a more comprehensive, accurate, and efficient assessment of the performance impact of new ATM solutions/concepts, thus having a positive impact across all the SESAR KPAs and ultimately on the achievement of the ATM Master Plan performance ambitions: Environment, Capacity, Cost Efficiency, Operational Efficiency, Safety and Security.
- Development of new techniques for better estimation of hidden variables, which provide new knowledge on airspace user preferences and constraints, helping understand the actual performance impact of new ATM concepts or solutions on airspace users.
- Development of new techniques for the comprehensive multi-scale classification of traffic patterns in the ECAC area, which enables an enhanced understanding of the performance impact of a certain operational concept under different traffic conditions, thus facilitating more accurate and comprehensive cost-benefit analyses.
- Introduction to active learning metamodelling techniques in ATM, which can be used to translate a complex ATM simulation model into a performance metamodel, improving computational tractability and interpretability of results by allowing a more efficient exploration of the simulator’s behaviour.
The SIMBAD Performance Modelling Framework facilitates a more comprehensive, accurate, and efficient assessment of the performance impact of new ATM solutions/concepts, thus having a positive impact across all the SESAR KPAs and ultimately on the achievement of the ATM Master Plan performance ambitions: Environment, Capacity, Cost Efficiency, Operational Efficiency, Safety and Security.