Periodic Reporting for period 1 - SYMBIOTIK (Context-aware adaptive visualizations for critical decision making)
Berichtszeitraum: 2022-10-01 bis 2023-09-30
We live in an era of information overload that impairs objective decision making, especially in time-sensitive contexts. Information Visualization (InfoVis) systems have been used to mitigate information overload, yet they have not yet unlocked their potential in critical decision-making scenarios. From emergency rooms and autonomous cars to operational command centres, a clear understanding and rapid assessment based on the available data can make the difference between life and death. SYMBIOTIK envisions an effortless interaction dialogue between human and InfoVis systems to support decision making processes, inspired by known biological principles and guided by artificial intelligence (AI). Critically, this dialogue requires AI solutions with context awareness, emotion sensing, and expressing capabilities. We propose a novel framework where both the human and the machine cooperate towards a common goal and evolve together. The overall objectives of the project are:
1. Support a natural dialogue between AI agents and humans following symbiosis principles.
2. Develop human-centric perception analysis methods and tools for situation/time-sensitive tasks in InfoVis systems.
3. Facilitate a novice-to-expert transition in InfoVis systems by means of context-aware self-adaptive graphical user interfaces.
Awareness principles will allow us to engineer complex systems, making them more resilient and more human-centric. We will define an integrative approach for awareness engineering and propose a specific open source implementation. Finally, we will demonstrate and validate the role and added-value of such an awareness framework in two scenarios: supporting novice-to-expert transitions and critical decision making. The awareness principles to be developed in this project can support learning, adaptation, and self-development of intelligent systems over long periods of time, not only in the InfoVis domain. Therefore, SYMBIOTIK has potential to achieve the real breakthroughs needed to bring awareness and emotional intelligence for decision-making tasks in computing systems. The results of the project will benefit a range of stakeholders, from human vision and brain researchers, computer scientists, citizens, as well as research funding bodies and policy makers.
1. Support a natural dialogue between AI agents and humans following symbiosis principles.
2. Develop human-centric perception analysis methods and tools for situation/time-sensitive tasks in InfoVis systems.
3. Facilitate a novice-to-expert transition in InfoVis systems by means of context-aware self-adaptive graphical user interfaces.
Awareness principles will allow us to engineer complex systems, making them more resilient and more human-centric. We will define an integrative approach for awareness engineering and propose a specific open source implementation. Finally, we will demonstrate and validate the role and added-value of such an awareness framework in two scenarios: supporting novice-to-expert transitions and critical decision making. The awareness principles to be developed in this project can support learning, adaptation, and self-development of intelligent systems over long periods of time, not only in the InfoVis domain. Therefore, SYMBIOTIK has potential to achieve the real breakthroughs needed to bring awareness and emotional intelligence for decision-making tasks in computing systems. The results of the project will benefit a range of stakeholders, from human vision and brain researchers, computer scientists, citizens, as well as research funding bodies and policy makers.
Based on D2.1 we have written a paper that reviews methodologies for assessing cognitive states, emphasizing the impact of states such as drowsiness, mental workload, stress, and mind wandering, and proposes strategies based on non-invasive brain imaging and peripheral physiological recordings. This achievement underscores our commitment to sharing our research outcomes with the scientific community (KPI4.3–4.6). Our analysis strategy involves a cross-modal neuroimaging approach, merging concurrently acquired data using electroencephalography (EEG), Electrodermal activity (EDA), electrocardiogram (ECG) and eye tracking. We performed a series of EEG+Eye pilot studies to investigate whether the presented stimuli would elicit measurable Event-Related Potentials (ERPs) on the scalp, since the physical characteristics of the visual stimuli are non-standard within the EEG literature. After this pilot, we went back to the chalkboard to change the task and the focus of the analysis drastically. Our new design will capitalize the well-established correlation between EEG signals and participants' cognitive workload. Our primary objective is to assess the impact of the Adaptation InfoVis design on both EEG classification accuracy and behavioral outcomes.
Deliverable D3.1 presented the initial version of our framework. A notation was introduced to express the involvement of agents in each adaptation stage, encompassing one or multiple end-users, one or multiple systems, including the symbiotic system, and one or multiple third parties. The framework was delineated as a specific instantiation derived from a comprehensive reference framework for intelligent UI adaptation. Deliverable D3.2 provided an in-depth description of the architectural design. We also identified various entities associated with the functional and non-functional requirements. Furthermore, we delved into the rationale behind the selection of specific technologies, programming languages, and methodologies. Deliverable D3.3 presented a preliminary validation of the SYMBIOTIK framework, focusing on the Symbiotik adaptation life cycle. Four experiments were conducted, covering all seven stages of the Symbiotik adaptation life cycle: goal, initiative, specification, application, transition, interpretation, and evaluation, as initially defined in D3.1. Each experiment addressed specific aspects of the life cycle, considering scenarios both inside and outside AEGIS AVT, with or without biosignals. The culmination involved achieving full coverage based on the latest version of the Adaptation panel.
WP4 activities focused on the planning and development of the first draft of the components that realize the mechanisms of the Continuous Adaptation Loop and the interactions with the Data Gathering and Annotating Layer (please check D5.1 section 3.1). Specifically, D4.1 (Adaptive visualization mechanism) demonstrates the current development status of the involved components and the communication pipelines between them for establishing a robust interaction between the end user and the SYMBIOTIK framework. The implementation of the adaptation requests generated by the SYMBIOTIK Adaptation Engine supports SYMBIOTIK’s main objective of delivering an AI-driven awareness framework to create symbiotic self-adaptive InfoVis systems. It is performed through the InfoVis Gateway which enhances the communication pipelines and streamlines the workflows for dashboard adaptation based on the Adaptation Panel configuration (developed in WP3). The Infovis Gateway ensures that the dashboard users receive timely updates based on the dashboard adaptation progress.
The outcomes of WP5's activities were reported in D5.1 (Integration setup and general specifications based on industrial case studies). We focused on the following achievements:
* Definition of requirements that are to be met by the SYMBIOTIK framework and other technical artefacts.
* Definition of the SYMBIOTIK architecture and of the subsequent component specifications.
* Definition of the Integration processes and the actual integration of the more mature components for testing and demonstration purposes
* Identification of the Use cases for the experimentation and validation of adaptation requests.
Regarding WP6, we have been thoroughly working on the communication and exploitation plan of SYMBIOTIK. Through D6.1 we have elaborated a Plan for Exploitation and Dissemination of Results (PEDR) on M12 that will be revised on M46. There, we have described strategies to engage stakeholders, promote the project and its achievements, as well as the knowledge generated in the project.
Finally, all ethics-related deliverables (D7.1–D7.3) were successfully submitted and WP8 activities (Portfolio) were initiated. The aim of the Pathfinder portfolio is to support the funded projects in achieving more efficiently and effectively their objectives, and more generally to enhance research, prepare transition to innovation and stimulate business opportunities, and strengthen the EIC Community.
Deliverable D3.1 presented the initial version of our framework. A notation was introduced to express the involvement of agents in each adaptation stage, encompassing one or multiple end-users, one or multiple systems, including the symbiotic system, and one or multiple third parties. The framework was delineated as a specific instantiation derived from a comprehensive reference framework for intelligent UI adaptation. Deliverable D3.2 provided an in-depth description of the architectural design. We also identified various entities associated with the functional and non-functional requirements. Furthermore, we delved into the rationale behind the selection of specific technologies, programming languages, and methodologies. Deliverable D3.3 presented a preliminary validation of the SYMBIOTIK framework, focusing on the Symbiotik adaptation life cycle. Four experiments were conducted, covering all seven stages of the Symbiotik adaptation life cycle: goal, initiative, specification, application, transition, interpretation, and evaluation, as initially defined in D3.1. Each experiment addressed specific aspects of the life cycle, considering scenarios both inside and outside AEGIS AVT, with or without biosignals. The culmination involved achieving full coverage based on the latest version of the Adaptation panel.
WP4 activities focused on the planning and development of the first draft of the components that realize the mechanisms of the Continuous Adaptation Loop and the interactions with the Data Gathering and Annotating Layer (please check D5.1 section 3.1). Specifically, D4.1 (Adaptive visualization mechanism) demonstrates the current development status of the involved components and the communication pipelines between them for establishing a robust interaction between the end user and the SYMBIOTIK framework. The implementation of the adaptation requests generated by the SYMBIOTIK Adaptation Engine supports SYMBIOTIK’s main objective of delivering an AI-driven awareness framework to create symbiotic self-adaptive InfoVis systems. It is performed through the InfoVis Gateway which enhances the communication pipelines and streamlines the workflows for dashboard adaptation based on the Adaptation Panel configuration (developed in WP3). The Infovis Gateway ensures that the dashboard users receive timely updates based on the dashboard adaptation progress.
The outcomes of WP5's activities were reported in D5.1 (Integration setup and general specifications based on industrial case studies). We focused on the following achievements:
* Definition of requirements that are to be met by the SYMBIOTIK framework and other technical artefacts.
* Definition of the SYMBIOTIK architecture and of the subsequent component specifications.
* Definition of the Integration processes and the actual integration of the more mature components for testing and demonstration purposes
* Identification of the Use cases for the experimentation and validation of adaptation requests.
Regarding WP6, we have been thoroughly working on the communication and exploitation plan of SYMBIOTIK. Through D6.1 we have elaborated a Plan for Exploitation and Dissemination of Results (PEDR) on M12 that will be revised on M46. There, we have described strategies to engage stakeholders, promote the project and its achievements, as well as the knowledge generated in the project.
Finally, all ethics-related deliverables (D7.1–D7.3) were successfully submitted and WP8 activities (Portfolio) were initiated. The aim of the Pathfinder portfolio is to support the funded projects in achieving more efficiently and effectively their objectives, and more generally to enhance research, prepare transition to innovation and stimulate business opportunities, and strengthen the EIC Community.