Periodic Reporting for period 2 - Dispatcher3 (Innovative processing for flight practices)
Berichtszeitraum: 2021-06-01 bis 2022-11-30
The primary objective of Dispatcher3 is to develop a prototype for the acquisition and preparation of historical flight data in order to give support on the optimisation of future flights providing predictive capabilities and advice to duty managers, dispatchers and pilots.
Dispatcher3 aims at improving airlines' pre-departure processes by providing an infrastructure able to leverage historical data and machine learning techniques to estimate the variability between planned and executed flight plans (achieving the objective of develop a tool with data engineering capabilities to clean, synchronise and merge past flight data with other operational environment data). These data were prepared so that analytic techniques could be applied (meeting another objective).
Airline's policies and target KPIs for pre-departure were identified (as indicated by one of the objectives), leading to the selection of the pre-departure indicators of expected holding and runway at arrival and total fuel consumption. In a larger prediction horizon (from D-1) indicators associated with ATFM delay were selected: probability of being regulated, location of the regulation (airspace or airport), probability of no delay if regulated, and distribution of expected delay if positive delay. These models were trained with data available at the indicated horizon achieving the objective of the project of predictive capabilities at different horizons.
The analysis of the features of the models enabled the team to identify precursors that impact these targeted KPIs (as requested by an additional objective of the project). The individual indicators were integrated into an advice generator to translate the predictions into actionable data (as defined in the objectives). This provides support to duty managers with information on the fleet status and advice on ATFM delays and reactionary delay (probability of breaching a curfew or missing an ATFM slot).
Dispatcher3 is composed of three layers: Data infrastructure, Predictive capabilities and Advice generator. The Data infrastructure is based on an Amazon Web Service cloud system and focuses on the preparation of data remaining generic enough to facilitate the exploitation of data for other future objectives. The Predictive capabilities layer is formed by two modules: data acquisition and preparation, and predictive models. Finally, the Advice generator provides a decision framework integrating the individual models into a dedicated dashboard and models.
All these components were validated with internal and external activities including workshops with an Advisory Board formed by airlines, experts and the Network Manager achieving the final objective of the project.
Dispatcher3 aims at improving airlines' pre-departure processes by providing an infrastructure able to leverage historical data and machine learning techniques to estimate the variability between planned and executed flight plans (achieving the objective of develop a tool with data engineering capabilities to clean, synchronise and merge past flight data with other operational environment data). These data were prepared so that analytic techniques could be applied (meeting another objective).
Airline's policies and target KPIs for pre-departure were identified (as indicated by one of the objectives), leading to the selection of the pre-departure indicators of expected holding and runway at arrival and total fuel consumption. In a larger prediction horizon (from D-1) indicators associated with ATFM delay were selected: probability of being regulated, location of the regulation (airspace or airport), probability of no delay if regulated, and distribution of expected delay if positive delay. These models were trained with data available at the indicated horizon achieving the objective of the project of predictive capabilities at different horizons.
The analysis of the features of the models enabled the team to identify precursors that impact these targeted KPIs (as requested by an additional objective of the project). The individual indicators were integrated into an advice generator to translate the predictions into actionable data (as defined in the objectives). This provides support to duty managers with information on the fleet status and advice on ATFM delays and reactionary delay (probability of breaching a curfew or missing an ATFM slot).
Dispatcher3 is composed of three layers: Data infrastructure, Predictive capabilities and Advice generator. The Data infrastructure is based on an Amazon Web Service cloud system and focuses on the preparation of data remaining generic enough to facilitate the exploitation of data for other future objectives. The Predictive capabilities layer is formed by two modules: data acquisition and preparation, and predictive models. Finally, the Advice generator provides a decision framework integrating the individual models into a dedicated dashboard and models.
All these components were validated with internal and external activities including workshops with an Advisory Board formed by airlines, experts and the Network Manager achieving the final objective of the project.
First, a workshop with the Advisory Board was carried out (09OCT20) followed up by bilateral discussions. Several meetings were held with Vueling to better understand their pre-departure operations and needs. These activities were reported in D1.1 - Technical resources and problem definition (01DEC20 - M7) reaching MS2 - Technical resources and problem definition completed.
Acquiring the datasets was a complex task due to difficulties on gathering the data from Vueling. Data Protection Agreements were put in place to facilitate the data transfer. D2.1 - Data definition and processing report (30APR21 - M11) was produced identifying data sources required for labelling and feature engineering and a roadmap for their acquisition; MS3 - Domain driven data engineering techniques identified was achieved.
The activities of WP3 - Data engineering and analytic techniques were extended to overlap the model development (WP4) and validation (WP5). D3.1 (22SEP22 - M28) describes the data, pipelines and the challenges of machine learning projects. With this MS5 - Domain driven analytic techniques identified was reached. In machine learning models development and validation are conducted interactively. An Extract-Load-Transform data pipeline was implemented for the definition and usage of a data lake. Two releases of Dispatcher3 were produced. The first release focused on the individual models targeting key performance indicators at two prediction horizons: pre-departure and planned flights. D4.1 - Technical documentation first release and D4.2 - Prototype package (first release) (27JUL22 - M26) formed this release (MS5 - First release results review). A workshop (followed by surveys) was then carried out with the Advisory Board (17MAY22 - M24). This, along a set of internal meetings and workshops, allowed the consortium to prioritise the further development of the models and the definition of the Advice Generator. The focus of the final release (D4.3 - Architecture and prototype description and D4.4 - Prototype package (final release) (17NOV22 - M30)) was on improving the models (ensuring data availability at the prediction horizon), the interface, and integration into high-level models to produce actionable advice.
The activities of WP5 - Prototype verification and validation were conducted in parallel to the models development once the Verification and Validation plan (D5.1) was submitted (11AUG21 - M15). The outcome of all these activities was presented in D5.2 - Verification and validation report (9DEC22 - M31); MS7 - Prototype verification and validation completed.
Steps towards industrialisation are summarised in D6.1 (13DEC22 - M31). Dispatcher3's components are suitable to be exploited independently. Pre-departure models could be incorporated into flight planning systems or crew support decision tools (such as Pilot3 (project 863802) or FPO cloud system by PACE). The pre-tactical (ATFM) and reactionary models could be integrated into support tools for duty managers.
Communication and dissemination activities include: the definition of the communication, dissemination and exploitation plan (D7.1 - 30NOV20 - M6), the launch of the project website, publication of 6 blog entries and 19 social media post (LinkedIn), participation in 7 conferences (papers and posters/videos) and industrial exhibitions. In addition a journal paper are under development. D7.2 (25NOV22 - M30) summarise these actions.
WP8 ensured the project management with D8.1 - Project management plan (30JUN20 - M1), D8.2 - Proof of signature of Consortium Agreement (04AUG20 - M3), 7 periodic monitoring reports and 2 Periodic Reports. The milestone MS1 - Project kick-off was reached at M1 (25JUN20), MS4 - Intermediate Review Meeting at M13 (7JUN21), and MS8 - Final acceptance with the finalisation of the action (close out meeting held 29NOV22 (M30)).
Acquiring the datasets was a complex task due to difficulties on gathering the data from Vueling. Data Protection Agreements were put in place to facilitate the data transfer. D2.1 - Data definition and processing report (30APR21 - M11) was produced identifying data sources required for labelling and feature engineering and a roadmap for their acquisition; MS3 - Domain driven data engineering techniques identified was achieved.
The activities of WP3 - Data engineering and analytic techniques were extended to overlap the model development (WP4) and validation (WP5). D3.1 (22SEP22 - M28) describes the data, pipelines and the challenges of machine learning projects. With this MS5 - Domain driven analytic techniques identified was reached. In machine learning models development and validation are conducted interactively. An Extract-Load-Transform data pipeline was implemented for the definition and usage of a data lake. Two releases of Dispatcher3 were produced. The first release focused on the individual models targeting key performance indicators at two prediction horizons: pre-departure and planned flights. D4.1 - Technical documentation first release and D4.2 - Prototype package (first release) (27JUL22 - M26) formed this release (MS5 - First release results review). A workshop (followed by surveys) was then carried out with the Advisory Board (17MAY22 - M24). This, along a set of internal meetings and workshops, allowed the consortium to prioritise the further development of the models and the definition of the Advice Generator. The focus of the final release (D4.3 - Architecture and prototype description and D4.4 - Prototype package (final release) (17NOV22 - M30)) was on improving the models (ensuring data availability at the prediction horizon), the interface, and integration into high-level models to produce actionable advice.
The activities of WP5 - Prototype verification and validation were conducted in parallel to the models development once the Verification and Validation plan (D5.1) was submitted (11AUG21 - M15). The outcome of all these activities was presented in D5.2 - Verification and validation report (9DEC22 - M31); MS7 - Prototype verification and validation completed.
Steps towards industrialisation are summarised in D6.1 (13DEC22 - M31). Dispatcher3's components are suitable to be exploited independently. Pre-departure models could be incorporated into flight planning systems or crew support decision tools (such as Pilot3 (project 863802) or FPO cloud system by PACE). The pre-tactical (ATFM) and reactionary models could be integrated into support tools for duty managers.
Communication and dissemination activities include: the definition of the communication, dissemination and exploitation plan (D7.1 - 30NOV20 - M6), the launch of the project website, publication of 6 blog entries and 19 social media post (LinkedIn), participation in 7 conferences (papers and posters/videos) and industrial exhibitions. In addition a journal paper are under development. D7.2 (25NOV22 - M30) summarise these actions.
WP8 ensured the project management with D8.1 - Project management plan (30JUN20 - M1), D8.2 - Proof of signature of Consortium Agreement (04AUG20 - M3), 7 periodic monitoring reports and 2 Periodic Reports. The milestone MS1 - Project kick-off was reached at M1 (25JUN20), MS4 - Intermediate Review Meeting at M13 (7JUN21), and MS8 - Final acceptance with the finalisation of the action (close out meeting held 29NOV22 (M30)).
Dispatcher3's models have shown their potential to provide an early identification of issues (e.g. flagging flights which might produce network-wide disruptions) so that actions can take place earlier, generating more predictable, reliable and efficient operations.
Dispatcher3's components could easily be isolated (individual models) facilitating their exploitation. The reactionary models provide a solid base to develop support systems for duty managers. These models might be suitable to identify situations in which pre-tactical intervention (e.g. aircraft swaps) are required.
Moreover, Dispatcher3 has contributed to advancing the state of the art on the usage of machine learning models in the aviation sector, identifying challenges and approaches for these type of projects. The architecture and methodology developed can be extended to other applications.
Dispatcher3's components could easily be isolated (individual models) facilitating their exploitation. The reactionary models provide a solid base to develop support systems for duty managers. These models might be suitable to identify situations in which pre-tactical intervention (e.g. aircraft swaps) are required.
Moreover, Dispatcher3 has contributed to advancing the state of the art on the usage of machine learning models in the aviation sector, identifying challenges and approaches for these type of projects. The architecture and methodology developed can be extended to other applications.