The hyPEr ExpeRt collaborative AI assistant

Projektinformationen

PEER

ID Finanzhilfevereinbarung: 101120406

DOI

10.3030/101120406

EK-Unterschriftsdatum 26 Juni 2023

Startdatum 1 Oktober 2023

Enddatum 30 September 2027

Finanziert unter

• Digital, Industry and Space

Gesamtkosten

€ 7 737 900,00

EU-Beitrag

€ 7 737 900,00

Koordiniert durch

VRIJE UNIVERSITEIT BRUSSEL
Belgium

Periodic Reporting for period 2 - PEER (The hyPEr ExpeRt collaborative AI assistant)

Berichtszeitraum: 2024-10-01 bis 2026-03-31

Zusammenfassung vom Kontext und den Gesamtzielen des Projekts

PEER project addresses the barriers to accepting AI in sequential decision-making problems, such as route planning, by systematically putting the user at the center of the entire AI design, development, deployment, and evaluation pipeline. To do this, PEER takes an interdisciplinary approach that considers both social sciences and artificial intelligence to facilitate a two-way communication flow to enhance user trust. PEER also develops an AI acceptance index for the evaluation of AI systems from a human-centric perspective and conducts integration and evaluation of these novel approaches in 4 real-world use cases.
PEER seeks to address the challenges of AI acceptance by systematically designing, realizing, and evaluating human-centric AI for sequential decision-making settings. The main objectives include: 1) To realize AI via design prototypes and to make the capabilities and limitations of AI known to stakeholders and users before the development process is carried out; 2) To improve feedback loops and enable a two-way communication flow between AI and human users to ensure better engagement in collaborative human-AI teamwork; 3) To measure progress towards truly mixed and trustworthy AI by defining qualitative and quantitative, transparent, reliable measurement scales and metrics for interactivity, acceptance, explainability and trustworthiness as well as perceived trust and fairness; 4) To integrate and evaluate the system taking into account a human-centric perspective, relying on the proposed indexes, next to more traditional performance indicators.
Besides, the PEER project takes a holistic approach involving technical sciences (AI), social sciences, and humanities (SSH) which will lead to novel ways of end-user engagement with AI and AI design. The project will focus on the end-user, and by design will consider intersectional factors (gender, ethnicity, age, socioeconomic status, disability) that could directly impact acceptance and trust in line with the EU's top agenda to achieve trustworthy AI. Furthermore, the efforts undertaken in the PEER initiative, addressing both social and technical dimensions, will serve as a reference for creating novel human-in-the-loop AI frameworks in Europe.

Arbeit, die ab Beginn des Projekts bis zum Ende des durch den Bericht erfassten Berichtszeitraums geleistet wurde, und die wichtigsten bis dahin erzielten Ergebnisse

From the project kick-off until the end of RP2, we have established the necessary structures and guidelines for efficient project management and made considerable progress in both foundational and technical areas for successful evaluation of the second Minimally Viable Product (MVP#2). A major focus of our work in the first year was the development of the methodology for gathering and analysing the sociotechnical requirements of the end users from our PEER use cases. These user-centered needs were carefully translated into concrete technical requirements whenever possible, ensuring that the AI systems we are developing are closely aligned with the values and practical needs of the users. Starting from sixth month, AI partners started with a catalogue of candidate AI algorithms relevant for each use cases. Through the analysis of the use cases through “tech cards”, we gained key insights into how the AI systems should function or enhanced in terms of collaboration and communication.

MVP#1 marked as the key milestone of PEER project, the initial functionalities of algorithms simulated using efficient Wizard-of-Oz approach (WoZ) with no real backend implementation. The result of such evaluation provided a tangible starting point for more advanced iterations of the system and real backend implementation.

After analysing the AI candidate algorithms applicable to relevant use cases, AI partners extended the classical AI Agent architecture and introduced a modular architecture and its building blocks. The primary motivation was to separate the agent’s components so that they could be developed independently by partners and customized for specific use cases. The advances in interactions and two-way communication of AI algorithms resulted in several contributions that perfectly fitted in the architecture of relevant use cases such as Accessible Route Navigation (AMS), Human-in-the-Loop Anomaly Detection (PDT), Personalized Indoor Shopping Assistance (SON), and Warehouse Configuration Management (DAT). Through the close collaboration between the AI, use case owners, and integration partners, the technical blueprint for the PEER system has been realized for the MVP#2 prototypes. The consortium successfully advanced by 65% the use case implementations to their MVP#2 phase. During this reporting period, the preliminary workflows were transitioned into fully functional, deployable microservice architectures based on the “PEER Common Blueprint”, marking a major transitioning from Wizard-of-Oz style prototypes to running systems that enable direct user interaction.

Building on top of the findings of the first iteration (MVP#1), the evaluation procedure, AI Acceptance Index and simulation framework were extended and developed further and will be tested with the PEER use-case partners in MVP#2 and iteratively updated until the end of project (MVP#3).

Fortschritte, die über den aktuellen Stand der Technik hinausgehen und voraussichtliche potenzielle Auswirkungen (einschließlich der bis dato erzielten sozioökonomischen Auswirkungen und weiter gefassten gesellschaftlichen Auswirkungen des Projekts)

In this RP2, several innovative developments have pushed our work beyond the current state of the art, particularly in the fields of:

Al Algorithms Adaptiveness and Communications

The project advances the state of the art in AI algorithms through fundamental research and cross-use-case innovation in explainable sequential decision-making and multi-objective reinforcement learning. It introduces novel approaches to enhance algorithm responsiveness by systematically integrating user feedback into the problem-solving loop, enabling personalized behaviour. Key contributions include preference-guided multi-objective optimization techniques that align algorithmic outcomes with user-defined priorities, as well as human-in-the-loop meta-level adaptation, illustrated in anomaly detection methods which reduce labelling task effort under limited labelled data. Furthermore, the project develops advanced user feedback mechanisms leveraging large language models to strengthen explainability, incorporating counterfactual explanation frameworks that provide transparent insights. Many of these advances are particularly demonstrated in relevant use cases and ready for user evaluations in the second (MVP#2).

Conceptualization of Collaborative AI Architecture and its Realization as “PEER Blueprint”

The project conceptualizes a modular and scalable framework for AI companions designed to address sequential decision-making problems while continuously aligning with human user needs and preferences. This architecture enables seamless collaboration between human and AI agents by embedding adaptability, personalization, and feedback-driven learning at its core. It is presented both as a generalizable design paradigm and through concrete instantiations tailored to specific use cases. Serving as the technical blueprint for system integration, the architecture underpins infrastructure deployment and architectural refactoring efforts for the MVP#2 prototypes. The foundational blueprint was successfully refined into a standardized, multi-tier microservice architecture applied consistently across all use cases.

PEER Simulation and Evaluation Framework:

A comprehensive evaluation procedure and simulation framework was developed which will be tested with the PEER use-case partners in MVP#2 evaluation phase. The AIA index, based on a user-centred trustworthiness framework of 8 constructs, has been implemented in the Peac²h platform for MVP#2 evaluation phase.