Periodic Reporting for period 2 - MENTOR (Machine LEarning in Optical NeTwORks)
Reporting period: 2023-01-01 to 2024-12-31
To date, optical fibre technologies have been a major driver of our societal progress as they enable global telecommunication and all high-capacity transmission systems. Optical networks have changed rapidly over the last decade, moving from fixed grid with direct-detection using C-band-only systems to current elastic optical networks with coherent technology and software defined networking. Now, C-band systems started being upgraded to C+L. This evolution is a consequence of an incoming exhaustion of the C-band due to the increase of bandwidth demand. The exploitation of bands beyond C is crucial and a promising technique to increase the throughput of optical networks.
The EID MENTOR has addressed the most relevant challenges of modern optical communication by combining ML&AI techniques and multiband transmission. Its objective has been to provide world class training for six early-stage researchers to become leaders in this sector.
Thanks to MENTOR’s consortium, we can access real datasets for network management and data monitoring, components, propagation impairments, and data traffic, essential for the training phase of algorithms.
Alongside Coordinator the Aston Institute of Photonic Technologies, Aston University, the consortium consists of the Polytechnic University of Catalonia (UPC), the Technical University of Denmark (DTU), Sant'Anna School of Advanced Studies (SSSA), Telecom Italia Mobile (TIM), Fibercop, Infinera Germany, Infinera Portugal and Orange.
The EID MENTOR has addressed the most relevant challenges of modern optical communication by combining ML&AI techniques and multiband transmission. Its objective has been to provide world class training for six early-stage researchers to become leaders in this sector.
Thanks to MENTOR’s consortium, we can access real datasets for network management and data monitoring, components, propagation impairments, and data traffic, essential for the training phase of algorithms.
Alongside Coordinator the Aston Institute of Photonic Technologies, Aston University, the consortium consists of the Polytechnic University of Catalonia (UPC), the Technical University of Denmark (DTU), Sant'Anna School of Advanced Studies (SSSA), Telecom Italia Mobile (TIM), Fibercop, Infinera Germany, Infinera Portugal and Orange.
MENTOR’s six researchers produced 11 peer-reviewed journal papers, 6 posters, 2 data sets, 1 piece of software and 24 conference papers. Here are our scientific highlights:
WP1
• Development of multi-mask learning algorithms to enhance the generalizability and performance of neural network-based equalizers for coherent optical systems.
• Pioneered parallelization techniques through knowledge distillation, enabling parallelizable processing for better recurrent neural network-based equalizers.
• Optimized hardware implementation to computational complexity and resource usage in FPGA platforms for neural network-based optical channel equalizers.
WP2
• Satellite-to-ground optical links bit error rate (BER) performance analysis over gamma-gamma distributed atmospheric time-variant weak-to-strong-turbulence channels.
• Optical turbulent time-variant channels performance monitoring using a simplified predictor based on Gaussian Process Regression (GPR).
• Backward-pumped and bidirectional-pumped Raman amplifiers numerical modeling in the C-band for practical Raman amplifiers performance evaluation.
• A physics-informed neural networks-based solver of Raman coupled differential equations.
• Using the MoleNetwork tool to generate backbone, metro core and metro core aggregation network topologies which satisfy the network settings and statistics provided by Telecom Italia Mobile.
WP3
• Data augmentation techniques to address data scarcity for training ML models
• Model-centric and data-centric approaches investigated to solve limited availability of failure data for training ML models, optimizing failure recovery process
• Reducing the computational complexity of neural networks, energy consumption and ML model running costs
• Data pre-processing to convert raw alarms to a ML-friendly form
• ML techniques for distinguishing relevant failure alarms from false alarms
• Novel RSA techniques based on Reinforcement Learning
WP4
• Development of:
o Algorithms and QoT estimation models based on optical constellations for failure detection, identification and severity estimation
o A multiband simulator
o Models and algorithms for lightpath provisioning
• Implementation of software to monitor and control QSFP28 form factor optical amplifiers.
• Proposing Digital Twin for multiband optical networks
• Methods for quality of transmission (QoT) estimation in the multiband system.
WP1
• Development of multi-mask learning algorithms to enhance the generalizability and performance of neural network-based equalizers for coherent optical systems.
• Pioneered parallelization techniques through knowledge distillation, enabling parallelizable processing for better recurrent neural network-based equalizers.
• Optimized hardware implementation to computational complexity and resource usage in FPGA platforms for neural network-based optical channel equalizers.
WP2
• Satellite-to-ground optical links bit error rate (BER) performance analysis over gamma-gamma distributed atmospheric time-variant weak-to-strong-turbulence channels.
• Optical turbulent time-variant channels performance monitoring using a simplified predictor based on Gaussian Process Regression (GPR).
• Backward-pumped and bidirectional-pumped Raman amplifiers numerical modeling in the C-band for practical Raman amplifiers performance evaluation.
• A physics-informed neural networks-based solver of Raman coupled differential equations.
• Using the MoleNetwork tool to generate backbone, metro core and metro core aggregation network topologies which satisfy the network settings and statistics provided by Telecom Italia Mobile.
WP3
• Data augmentation techniques to address data scarcity for training ML models
• Model-centric and data-centric approaches investigated to solve limited availability of failure data for training ML models, optimizing failure recovery process
• Reducing the computational complexity of neural networks, energy consumption and ML model running costs
• Data pre-processing to convert raw alarms to a ML-friendly form
• ML techniques for distinguishing relevant failure alarms from false alarms
• Novel RSA techniques based on Reinforcement Learning
WP4
• Development of:
o Algorithms and QoT estimation models based on optical constellations for failure detection, identification and severity estimation
o A multiband simulator
o Models and algorithms for lightpath provisioning
• Implementation of software to monitor and control QSFP28 form factor optical amplifiers.
• Proposing Digital Twin for multiband optical networks
• Methods for quality of transmission (QoT) estimation in the multiband system.
WP1 AIPT
Significant progress has been made in neural network-based equalization for coherent optical communication systems. ESR1 worked on leveraging multi-task learning (MTL) to enhance the generalizability of neural network equalizers, enabling a single model to adapt to varying transmission conditions without the need for re-training. Q-factor improved by up to 4 dB compared to conventional digital signal processing techniques. We also developed the parallelization of recurrent neural network equalizers via knowledge distillation for faster and more efficient data processing, useful for technologies such as 6G. Enhanced connectivity and good data transmission is crucial for efficient digital infrastructure and the proliferation of data-intensive applications.
WP2 DTU
Of note is the new neural networks-based solver in the C-band with backward-pumped and bidirectional-pumped schemes. The outlook for the next generation of fiber-optic communication systems operating over ultra-wide band, Raman amplifiers are attractive solutions due to their broad bandwidth, and the ability to provide arbitrary gain profiles in a controlled way. Crucial to optimizing Raman amplifiers over large bandwidths is the proper allocation of pump powers and wavelengths. The new neural networks-based solver could be used to foresee the impact of Stimulated Raman Scattering on the fiber span of a C+L or S+C+L-band system and to design the distributed Raman amplifiers to compensate for this effect. Also, this new neural networks-based solver could be used to optimize wide-bandwidth (C+L- or S+C+L-band) Raman amplifiers to properly select multiple pump wavelengths and powers for desired gain profiles. It is an effective way for Raman amplifier modelling and performance evaluation before the real fiber Raman amplifiers are fabricated.
WP3 SSSA
Computational complexity reduction of ML models will impact not just optical networks but the world of AI. AI/ML-based data centres consume one order of magnitude more power than traditional data centres which means that larger ML models consume high power. Our work on computational complexity reduction of ML models can be used to minimize the carbon footprint, analysing and predicting equipment failures, vendor-agnostic solutions for alarm management, and developing ML-based algorithms for failure prediction; there are contributors in maintaining a stable network operation, minimizing downtime. The potential here is to revolutionize optical network management, reducing operational costs and enhancing service provisioning. This contributes to advancing automation and optimization technologies.
WP4 UPC
Alongside failure management, WP4 has worked on the OCATA optical layer digital twin and investigated its applications e.g. light path provisioning and optical amplifier control. These applications if verified will enable effective network optimization, accurate failure management, thereby reducing the network margins and loads. The societal implications would consist in a more efficient and sustainable infrastructure supporting Internet. WP4 has also worked on Adaptive Neural Network-based models for C+L+S band scenario. The proposed adaptive neural network-based model is able to provide a more efficient and accurate way to handle highly non-linear optical network transmission. MB optical transmission is considered the next technological evolution in optical WDM transport capable of dealing with 5G and beyond in a cost-effective manner. Hence, a revolution to optical network management.
Significant progress has been made in neural network-based equalization for coherent optical communication systems. ESR1 worked on leveraging multi-task learning (MTL) to enhance the generalizability of neural network equalizers, enabling a single model to adapt to varying transmission conditions without the need for re-training. Q-factor improved by up to 4 dB compared to conventional digital signal processing techniques. We also developed the parallelization of recurrent neural network equalizers via knowledge distillation for faster and more efficient data processing, useful for technologies such as 6G. Enhanced connectivity and good data transmission is crucial for efficient digital infrastructure and the proliferation of data-intensive applications.
WP2 DTU
Of note is the new neural networks-based solver in the C-band with backward-pumped and bidirectional-pumped schemes. The outlook for the next generation of fiber-optic communication systems operating over ultra-wide band, Raman amplifiers are attractive solutions due to their broad bandwidth, and the ability to provide arbitrary gain profiles in a controlled way. Crucial to optimizing Raman amplifiers over large bandwidths is the proper allocation of pump powers and wavelengths. The new neural networks-based solver could be used to foresee the impact of Stimulated Raman Scattering on the fiber span of a C+L or S+C+L-band system and to design the distributed Raman amplifiers to compensate for this effect. Also, this new neural networks-based solver could be used to optimize wide-bandwidth (C+L- or S+C+L-band) Raman amplifiers to properly select multiple pump wavelengths and powers for desired gain profiles. It is an effective way for Raman amplifier modelling and performance evaluation before the real fiber Raman amplifiers are fabricated.
WP3 SSSA
Computational complexity reduction of ML models will impact not just optical networks but the world of AI. AI/ML-based data centres consume one order of magnitude more power than traditional data centres which means that larger ML models consume high power. Our work on computational complexity reduction of ML models can be used to minimize the carbon footprint, analysing and predicting equipment failures, vendor-agnostic solutions for alarm management, and developing ML-based algorithms for failure prediction; there are contributors in maintaining a stable network operation, minimizing downtime. The potential here is to revolutionize optical network management, reducing operational costs and enhancing service provisioning. This contributes to advancing automation and optimization technologies.
WP4 UPC
Alongside failure management, WP4 has worked on the OCATA optical layer digital twin and investigated its applications e.g. light path provisioning and optical amplifier control. These applications if verified will enable effective network optimization, accurate failure management, thereby reducing the network margins and loads. The societal implications would consist in a more efficient and sustainable infrastructure supporting Internet. WP4 has also worked on Adaptive Neural Network-based models for C+L+S band scenario. The proposed adaptive neural network-based model is able to provide a more efficient and accurate way to handle highly non-linear optical network transmission. MB optical transmission is considered the next technological evolution in optical WDM transport capable of dealing with 5G and beyond in a cost-effective manner. Hence, a revolution to optical network management.