AI-Ops Framework for Automated, Intelligent and Reliable Data/AI Pipelines Lifecycle with Humans-in-the-Loop and Coupling of Hybrid Science-Guided and AI Models

Informations projet

AI-DAPT

N° de convention de subvention: 101135826

DOI

10.3030/101135826

Date de signature de la CE 23 Octobre 2023

Date de début 1 Janvier 2024

Date de fin 30 Juin 2027

Financé au titre de

• Digital, Industry and Space

Coût total

€ 8 995 540,00

Contribution de l’UE

€ 8 995 540,00

Coordonné par

ATHINA-EREVNITIKO KENTRO KAINOTOMIAS STIS TECHNOLOGIES TIS PLIROFORIAS, TON EPIKOINONION KAI TIS GNOSIS
Greece

Periodic Reporting for period 2 - AI-DAPT (AI-Ops Framework for Automated, Intelligent and Reliable Data/AI Pipelines Lifecycle with Humans-in-the-Loop and Coupling of Hybrid Science-Guided and AI Models)

Période du rapport: 2025-03-01 au 2026-04-30

Résumé du contexte et des objectifs généraux du projet

AI-DAPT aims at reinstating the pure data-related work in its rightful place in AI and at reinforcing the generalizability, reliability, trustworthiness and fairness of Al solutions. AI-DAPT brings forward a data-centric mentality in AI, that is effectively fused with a model-centric, science-based approach, across the complete lifecycle of AI-Ops, by introducing end-to-end automation that is supported by AI-based techniques, across the design, the execution, the observability and the constant optimisation of intelligent data/AI pipelines.
In its quest for automation, AI-DAPT will design a novel AI-Ops / intelligent pipeline lifecycle framework cross-cutting the different business, legal/ethics, data, AI logic/models, and system requirements while always ensuring a human-in-the-loop (HITL) approach. Taking into consideration the dual “data and model” design, development and operation perspectives that need to be gracefully and effectively brought together, AI-DAPT will establish the underlying methodological and technical foundations across different axes that will complementarily and interactively work together as follows: A. AI/Data Axis (addressing Challenges 1-7). AI-DAPT shall adopt novel automated approaches, fused with targeted human-in-the-loop aspects, to improve “data for AI” pipelines in a systematic and scalable way. Typical data management activities including (but not limited to) data definition, preparation, annotation, cleaning, manipulation, synthetic generation and observability will be revamped through AI-driven automation that further leverages Explainable AI (XAI) techniques to ensure human interaction and informed intervention, whenever required for taking the final decisions regarding the pipeline configuration, as well as for ensuring the quality of the data and the ethical use of the underlying AI. Through the AI-DAPT data-centric AI research, the raw datasets will be revamped into appropriate, added value, up-to-date and reusable features, that effectively reduce the “time to insights”; B. AI/Model Axis (addressing Challenges 8-11). AI-DAPT shall explore and promote hybrid science-AI solutions, bringing together data-driven AI models and science-based first-principles models, that build on high-quality and reliable data. AI-DAPT practically introduces targeted automation interventions on the AI model building, training, validation and observability steps, that shall allow continuous, dynamic AI improvements by diminishing the “time to detection” and “time to resolution” for any AI pipeline problem, adjusting the training on-the-fly and ensuring that AI will work reliably well across diverse production environments and settings.
In order to demonstrate the actual innovation and added value that can be derived through the AI-DAPT scientific advancements, the AI-DAPT results will be validated in two, interlinked axes: I. Through their actual application to address real-life problems in four (4) representative industries that are characterized by a varying degree of AI maturity: (a) Health, (b) Robotics, (c) Energy, and (d) Manufacturing; II. Through their integration in different AI solutions, either open source (e.g. Jupyter, Acumos AI) or commercial (S5 Enterprise Analytics Suite, Qlik, etc.). The purpose of such an integration is to demonstrate that the AI-DAPT results bring added value within the established AI market landscape.

Travail effectué depuis le début du projet jusqu’à la fin de la période considérée dans le rapport et principaux résultats atteints jusqu’à présent

Scientific and Technical Objectives (STO)
STO.1: To set and implement the underlying foundations for data-centric trustworthy AI solutions through reliable and interoperable pipelines that consistently automate the end-to-end data/AI management processes and effectively fuse first-principles scientific models and data-driven AI/machine learning models.
Our main achievements are:
• Refinement and consolidation of user and technical requirements across the four demonstrators.
• Further development of the AI-DAPT Data and AI Pipeline Lifecycle Framework, providing a common basis for data-centric and trustworthy AI workflows.
• Application of ethics-by-design and data protection assessment methodologies throughout the technical work.
• Continuous monitoring of emerging technologies through the AI Technology Radar and alignment of the project research agenda with demonstrator needs.
• Definition of a common framework supporting the combination of domain knowledge, scientific models and machine learning techniques.
STO.2: To design and deliver appropriate automated and XAI-based “Data for AI” techniques cross-cutting the data pipeline lifecycle operations, from data mining/harvesting, investigation, documentation, annotation, and cleaning to fit-for-purpose data valuation and synthetic data generation, to allow accruing the right value for the right data at the right time.
Our main achievements are:
• Development of services supporting data harvesting, documentation, valuation, cleaning, feature engineering and synthetic data generation.
• Introduction of common APIs and metadata structures enabling interoperability across the different Data-for-AI services.
• Availability of these services through the AI-DAPT Alpha Platform Release, allowing their use in demonstrator workflows.
• Initial use of the developed services on real datasets from the Health, Energy, Manufacturing and Robotics pilots.
• Creation of the first reusable and traceable data preparation workflows that reduce the effort required to build AI-ready datasets.
STO.3: To develop rigorous automated, fair and trusted “Hybrid Science-guided AI Models” techniques cross-cutting the AI pipeline lifecycle operations, from interactive training, explainability and evaluation to continuous delivery and observability to provide constantly accurate, data-driven and scientifically consistent insights in an energy-efficient manner.
Our main achievements are:
• Progress in hybrid AI approaches that combine machine learning with domain-specific knowledge and scientific models.
• Incorporation of explainability and experiment tracking capabilities into the AI-DAPT Alpha Platform Release.
• Preliminary evaluation results indicating the feasibility of the proposed hybrid and explainable AI approaches in the targeted industrial domains.
STO.4: To serve automated, end-to-end data/AI pipelines through the novel, integrated AI-DAPT AI-Ops platform based on easily deployable and scalable services, enabling efficient, trustful, reliable and interoperable exchanges with a wealth of sources, systems and data spaces.
Our main achievements are:
• Delivery of the AI-DAPT Reference Architecture and the AI-DAPT Alpha Platform Release.
• Integration of Data-for-AI and Models-for-AI services within a common AI-Ops environment.
• Deployment of secure and scalable infrastructures supporting both development and operational testing activities.
• Introduction of continuous integration, deployment and monitoring mechanisms to support platform evolution.
• Successful onboarding of demonstrator datasets and execution of end-to-end workflows through the AI-DAPT Alpha Platform Release.

Progrès au-delà de l’état des connaissances et impact potentiel prévu (y compris l’impact socio-économique et les conséquences sociétales plus larges du projet jusqu’à présent)

Till now, the scientific impact that the AI-DAPT consortium has achieved so far is reflected in 2 journal papers (+1 accepted), 9 conf. papers (+5 accepted), 3 book chapters, 2 pre-prints, 1 poster.