Periodic Reporting for period 2 - TraDE-OPT (Training Data-driven Experts in OPTimization)
Reporting period: 2022-06-01 to 2024-11-30
The TraDE-OPT project aimed to train 15 experts in optimization for data science, equipping them with a strong multidisciplinary background to advance the state of the art.
In today's world, data is generated at an unprecedented scale. Extracting meaningful insights from vast and complex datasets is one of the central challenges of data science. Optimization plays a fundamental role in this process, providing the theoretical and algorithmic foundations for solving complex problems efficiently. While recent advances in optimization, machine learning, functional analysis, and signal processing have led to powerful mathematical tools and algorithms, many challenges remain. The size, heterogeneity, and incompleteness of modern datasets continue to pose significant obstacles, requiring new and more efficient approaches.
TraDE-OPT addressed these challenges by focusing on exploiting structure—whether in data, models, or computational platforms—to develop advanced optimization algorithms with guaranteed performance. These methods leveraged decomposition techniques, incremental and stochastic strategies, and parallel or distributed computing, significantly improving scalability and efficiency in real-world applications.
To achieve these goals, TraDE-OPT provided an innovative training program, combining strong technical foundations with essential employability skills, such as entrepreneurship, communication, and career planning.
As a result, TraDE-OPT fellows are now well-prepared for outstanding careers in both academia and industry, contributing to the continued advancement of optimization and data science for the benefit of society.
In today's world, data is generated at an unprecedented scale. Extracting meaningful insights from vast and complex datasets is one of the central challenges of data science. Optimization plays a fundamental role in this process, providing the theoretical and algorithmic foundations for solving complex problems efficiently. While recent advances in optimization, machine learning, functional analysis, and signal processing have led to powerful mathematical tools and algorithms, many challenges remain. The size, heterogeneity, and incompleteness of modern datasets continue to pose significant obstacles, requiring new and more efficient approaches.
TraDE-OPT addressed these challenges by focusing on exploiting structure—whether in data, models, or computational platforms—to develop advanced optimization algorithms with guaranteed performance. These methods leveraged decomposition techniques, incremental and stochastic strategies, and parallel or distributed computing, significantly improving scalability and efficiency in real-world applications.
To achieve these goals, TraDE-OPT provided an innovative training program, combining strong technical foundations with essential employability skills, such as entrepreneurship, communication, and career planning.
As a result, TraDE-OPT fellows are now well-prepared for outstanding careers in both academia and industry, contributing to the continued advancement of optimization and data science for the benefit of society.
The TraDE-OPT network brought together 15 Early Stage Researchers (ESRs) across seven universities and one SME in seven different countries, creating a rich and diverse research environment at the intersection of optimization, data science, and machine learning.
To provide innovative training, we organized three schools, one workshop and one final conference, offering a comprehensive scientific and soft-skills program. The scientific sessions covered convex optimization, inverse problems, machine learning, industry presentations, an algorithmic bootcamp, a TRIZ course, equipping fellows with cutting-edge tools to tackle complex optimization problems. The soft-skills training focused on entrepreneurship, scientific writing, communication, CV writing and job market preparation, EU proposal writing, ensuring that ESRs were well-prepared for careers in both academia and industry.
The ESRs actively engaged with the global research community, presenting their findings at approximately 50 events, including conferences, workshops, and invited seminars. Their research has already resulted in around 50 publications in respected journals and international conferences, demonstrating the project’s contribution to the field. Beyond academic contributions, the TraDE-OPT consortium also prioritized public outreach, sharing its work with broader audiences to promote interest in optimization and its practical applications.
One of the major achievements of TraDE-OPT is the focus on exploiting problem structures in optimization models. Researchers developed new first-order algorithms that take advantage of structural properties to improve convergence for convex and nonconvex problems. These algorithms incorporate decomposition and parallelization techniques, making them scalable to high-dimensional data, which is essential for modern data science applications. Additionally, new randomized and incremental methods were introduced to improve computational efficiency and robustness. Machine learning techniques were integrated into optimization frameworks through unrolling techniques and hyperparameter tuning, linking data-driven models with mathematical optimization.
A key part of the project was collaboration with industry, particularly through secondments, where fellows worked directly with companies to refine their theoretical contributions and transfer optimization techniques to real-world applications. Advances in imaging science led to the development of new variational approaches for image reconstruction and segmentation, which could improve medical imaging techniques such as MRI reconstruction. Optimization methods from TraDE-OPT were successfully applied to chromatography in analytical chemistry, enhancing data processing and experimental analysis. Additionally, algorithms developed within the project contributed to the identification of railway vehicle components’ characteristics from measured data, supporting system monitoring and predictive maintenance in transportation engineering.
Through innovative research and training, international collaboration, and industry partnerships, TraDE-OPT equipped a new generation of researchers with essential skills. By integrating cutting-edge optimization techniques with interdisciplinary applications, TraDE-OPT has significantly advanced the state of the art in data-driven optimization.
To provide innovative training, we organized three schools, one workshop and one final conference, offering a comprehensive scientific and soft-skills program. The scientific sessions covered convex optimization, inverse problems, machine learning, industry presentations, an algorithmic bootcamp, a TRIZ course, equipping fellows with cutting-edge tools to tackle complex optimization problems. The soft-skills training focused on entrepreneurship, scientific writing, communication, CV writing and job market preparation, EU proposal writing, ensuring that ESRs were well-prepared for careers in both academia and industry.
The ESRs actively engaged with the global research community, presenting their findings at approximately 50 events, including conferences, workshops, and invited seminars. Their research has already resulted in around 50 publications in respected journals and international conferences, demonstrating the project’s contribution to the field. Beyond academic contributions, the TraDE-OPT consortium also prioritized public outreach, sharing its work with broader audiences to promote interest in optimization and its practical applications.
One of the major achievements of TraDE-OPT is the focus on exploiting problem structures in optimization models. Researchers developed new first-order algorithms that take advantage of structural properties to improve convergence for convex and nonconvex problems. These algorithms incorporate decomposition and parallelization techniques, making them scalable to high-dimensional data, which is essential for modern data science applications. Additionally, new randomized and incremental methods were introduced to improve computational efficiency and robustness. Machine learning techniques were integrated into optimization frameworks through unrolling techniques and hyperparameter tuning, linking data-driven models with mathematical optimization.
A key part of the project was collaboration with industry, particularly through secondments, where fellows worked directly with companies to refine their theoretical contributions and transfer optimization techniques to real-world applications. Advances in imaging science led to the development of new variational approaches for image reconstruction and segmentation, which could improve medical imaging techniques such as MRI reconstruction. Optimization methods from TraDE-OPT were successfully applied to chromatography in analytical chemistry, enhancing data processing and experimental analysis. Additionally, algorithms developed within the project contributed to the identification of railway vehicle components’ characteristics from measured data, supporting system monitoring and predictive maintenance in transportation engineering.
Through innovative research and training, international collaboration, and industry partnerships, TraDE-OPT equipped a new generation of researchers with essential skills. By integrating cutting-edge optimization techniques with interdisciplinary applications, TraDE-OPT has significantly advanced the state of the art in data-driven optimization.
The TraDE-OPT project has significantly advanced the field of data-driven optimization by developing new mathematical frameworks, efficient optimization algorithms, and scalable computational tools. The project has successfully bridged theoretical advancements with real-world applications, tackling large-scale convex and non-convex optimization problems relevant to machine learning, inverse problems, and signal processing. Through novel parallel and incremental algorithms, it improved stochastic and deterministic optimization methods, and provided theoretical guarantees for new optimization techniques. One of its most significant contributions has been the design and numerical testing of advanced algorithms for industrial applications, including applications in MRI reconstruction, identification of railway vehicle components’ characteristics from measured data, analytical chemistry, and energy market optimization.
The project has had substantial socio-economic and societal impacts, notably through its highly structured PhD training program, which has provided 15 early-stage researchers (ESRs) with cutting-edge expertise in optimization, machine learning, and computational mathematics. The project has actively engaged with the wider research community through numerous conferences, workshops, and open-access publications, facilitating the dissemination and adoption of its findings.
Furthermore, TraDE-OPT has fostered industry-academic collaborations, making its results applicable to real-world challenges such as high-speed data processing, automated decision-making, and resource-efficient AI models. The open-source nature of its software and algorithms ensures that companies and research institutions can further develop and integrate these optimization tools into emerging technologies.
In terms of broader societal engagement, the project has promoted STEM outreach through participation in science festivals, workshops for children and teenagers, and public engagement events.
In summary, TraDE-OPT has pushed the boundaries of optimization research, trained a next generation of experts, and laid the groundwork for long-term impact in both academia and industry. Its contributions will continue to influence the development of efficient and scalable optimization algorithms, driving innovation across various scientific and technological domains.
The project has had substantial socio-economic and societal impacts, notably through its highly structured PhD training program, which has provided 15 early-stage researchers (ESRs) with cutting-edge expertise in optimization, machine learning, and computational mathematics. The project has actively engaged with the wider research community through numerous conferences, workshops, and open-access publications, facilitating the dissemination and adoption of its findings.
Furthermore, TraDE-OPT has fostered industry-academic collaborations, making its results applicable to real-world challenges such as high-speed data processing, automated decision-making, and resource-efficient AI models. The open-source nature of its software and algorithms ensures that companies and research institutions can further develop and integrate these optimization tools into emerging technologies.
In terms of broader societal engagement, the project has promoted STEM outreach through participation in science festivals, workshops for children and teenagers, and public engagement events.
In summary, TraDE-OPT has pushed the boundaries of optimization research, trained a next generation of experts, and laid the groundwork for long-term impact in both academia and industry. Its contributions will continue to influence the development of efficient and scalable optimization algorithms, driving innovation across various scientific and technological domains.