Periodic Reporting for period 1 - TEAMING.AI (Human-AI Teaming Platform for Maintaining and Evolving AI Systems in Manufacturing)
Reporting period: 2021-01-01 to 2022-06-30
Smart Manufacturing plays a critical role in maintaining the competitiveness of companies and organizations, by supporting them at different levels such as process optimization, resource efficiency, predictive maintenance, and quality control. Nevertheless, current AI technologies that are rapidly penetrating industrial sectors at those levels remain essentially narrow AI systems. This is due to the lack of self-adaptation in the AI’s capability to assimilate and interpret new information outside of its predefined programmed parameters and existing computational models are only applicable in a limited setting so far. However, a key challenge and research gap is the modeling of dynamic process factors and temporal constraints that are implicated in managing uncertainty in task progress, interactions, and communication. Thus, our focus in TEAMING.AI is on situation awareness of human-AI dependencies in dynamic settings (see Fig. 1).
In the TEAMING.AI project we aim to develop a human-AI teaming framework that integrates the strengths of both, the flexibility of human intelligence and the scale-up capability of machine intelligence. The envisioned TEAMING.AI platform has the goal to orchestrate the information exchange and to organize the collected information within a layered knowledge graph, reduce the information to its key aspects and semantically enrich this knowledge with context information. Transparent storage and processing of information is the foundation for a decision support system that can be understood and further analyzed by human team members. In TEAMING.AI we investigate to what extent a knowledge graph (KG) can be used as a central component in an enhanced ML system as a basis for a situation awareness model with update and reasoning mechanisms that allow for a dynamic contextualization of manufacturing settings including local situations, tasks, and organizational context.
In the TEAMING.AI project we aim to develop a human-AI teaming framework that integrates the strengths of both, the flexibility of human intelligence and the scale-up capability of machine intelligence. The envisioned TEAMING.AI platform has the goal to orchestrate the information exchange and to organize the collected information within a layered knowledge graph, reduce the information to its key aspects and semantically enrich this knowledge with context information. Transparent storage and processing of information is the foundation for a decision support system that can be understood and further analyzed by human team members. In TEAMING.AI we investigate to what extent a knowledge graph (KG) can be used as a central component in an enhanced ML system as a basis for a situation awareness model with update and reasoning mechanisms that allow for a dynamic contextualization of manufacturing settings including local situations, tasks, and organizational context.
Technically, our focus in the first reporting period (M1-M18) was therefore concentrated on the development of concepts, methods, and software tools that help us to leverage a KG for the above-described vision. This includes:
• Monitoring and tracking w.r.t the KG model to extract and match evidence cues, e.g. for task assignment between humans and AI („trust model“)
• Extension to a multi-layer KG to encode also high-level policies, e.g. safety and ergonomic guidelines („auditable ethics“)
• Utilizing KG embeddings for reasoning about the situation awareness model, e.g. uncertainty model about consistency or incompleteness („self-diagnosis“)
• Novel techniques for the synchronization between KG and embeddings on it („KG dynamics“)
In particular, we made concentrated efforts towards the following aspects:
The Knowledge Graph as Model for Situation Awareness
Development of a layered, modular knowledge graph to support teaming scenarios and the shared decision making within. The design of KG focused on the representation of dynamic and evolving knowledge, particularly in the context of processes. The setup and implementation of the KG follows a knowledge graph management lifecycle (see Fig. 2, left) that structures KG population activities as well as the monitoring of the KG once it is deployed. We developed components for orchestrating dynamic human AI tasks assignment and keeping the KG synchronized with its embedding space, so that changes in the KG can be immediately applied to the embeddings of the affected nodes to handle downstream relational machine learning tasks.
The Teaming Intelligence Aspect of Human-AI Collaboration
In TEAMING.AI the human-AI teaming is centered on a coordination mechanism that supports teaming intelligence for human-AI collaboration by means of operationalization of the 4S framework based on the interdependence analysis of teamwork. We developed concepts for its technical realization in the use cases. The main components are the Teaming Model based on the Dynamic KG and its sub-meta-models for representing and modeling activities, events, processes, and high-level policies as basis for synchronizing and aligning human-AI communication, reciprocal learning (AI assists human in decision making and, vice versa, KG enhancement by humans) and task assignment between humans and AI.
Development and Integration of Components for a Proof of Concept
The development of the Use Cases started with a requirement engineering and the description of user stories that will be used as anchor points for validation tests of teaming scenarios. The requirement engineering included an importance evaluation of an extended list software quality attributes (scalability, robustness) that also included attributes towards teaming aspects (trustworthiness, explicability). Based on this assessment, a reference software architecture for the TEAMING.AI platform has been developed that has a special focus on scalability and timing requirements of ML components and the KG runtime layer (see Fig. 3).
• Monitoring and tracking w.r.t the KG model to extract and match evidence cues, e.g. for task assignment between humans and AI („trust model“)
• Extension to a multi-layer KG to encode also high-level policies, e.g. safety and ergonomic guidelines („auditable ethics“)
• Utilizing KG embeddings for reasoning about the situation awareness model, e.g. uncertainty model about consistency or incompleteness („self-diagnosis“)
• Novel techniques for the synchronization between KG and embeddings on it („KG dynamics“)
In particular, we made concentrated efforts towards the following aspects:
The Knowledge Graph as Model for Situation Awareness
Development of a layered, modular knowledge graph to support teaming scenarios and the shared decision making within. The design of KG focused on the representation of dynamic and evolving knowledge, particularly in the context of processes. The setup and implementation of the KG follows a knowledge graph management lifecycle (see Fig. 2, left) that structures KG population activities as well as the monitoring of the KG once it is deployed. We developed components for orchestrating dynamic human AI tasks assignment and keeping the KG synchronized with its embedding space, so that changes in the KG can be immediately applied to the embeddings of the affected nodes to handle downstream relational machine learning tasks.
The Teaming Intelligence Aspect of Human-AI Collaboration
In TEAMING.AI the human-AI teaming is centered on a coordination mechanism that supports teaming intelligence for human-AI collaboration by means of operationalization of the 4S framework based on the interdependence analysis of teamwork. We developed concepts for its technical realization in the use cases. The main components are the Teaming Model based on the Dynamic KG and its sub-meta-models for representing and modeling activities, events, processes, and high-level policies as basis for synchronizing and aligning human-AI communication, reciprocal learning (AI assists human in decision making and, vice versa, KG enhancement by humans) and task assignment between humans and AI.
Development and Integration of Components for a Proof of Concept
The development of the Use Cases started with a requirement engineering and the description of user stories that will be used as anchor points for validation tests of teaming scenarios. The requirement engineering included an importance evaluation of an extended list software quality attributes (scalability, robustness) that also included attributes towards teaming aspects (trustworthiness, explicability). Based on this assessment, a reference software architecture for the TEAMING.AI platform has been developed that has a special focus on scalability and timing requirements of ML components and the KG runtime layer (see Fig. 3).
Operationalizing Teaming Intelligence Concepts from the Social Sciences
The utilization of knowledge modeling techniques by means of knowledge graphs to analyze and model the teaming interdependence relationships along the team and task structure addresses the challenges outlined for situation awareness and its reasoning within the model and about it. The introduced concept of abstract activities as a mechanism to model performer/supporter team patterns leads to new ways to describe teaming processes and activities as well as teaming events that trigger interactions.
Knowledge Graph Modularization
Advances in digital and manufacturing technologies are strongly driven by data as a key enabler, creating opportunities beyond classic monitoring and predictive applications. In this context, KGs have strong application potential to create an integrated multi-perspective machine data space from heterogeneous data silos, lift and contextualize machine generated data, and facilitate cooperation between various domain experts and AI agents based on shared concepts. A key benefit of the proposed approach is that the resulting KGs support navigation across abstraction hierarchies, enabling bottom-up contextualization of raw data on the one hand, and top-down explanations by linking to lower levels of granularity on the other hand.
Dynamic Knowledge Graph Embeddings
Graph embeddings allow the use of graph-structured data for applications that, by definition, rely on numerical feature vectors as inputs. In TEAMING.AI the transformation of KGs into sets of numerical feature vectors is performed by embedding algorithms, which map the elements of the graph into a low-dimensional embedding space. However, these methods mostly assume a static knowledge graph, so subsequent updates inevitably require a re-run of the embedding process. In Teaming.AI we developed the NaviPy Approach which aims to maintain advantages of established embedding methods while making them accessible to dynamic domains. Relational Graph Convolutional Networks are adapted for efficient reconstructing node embeddings based solely on local neighborhoods. Preliminary results suggest that the performance of successive machine learning tasks is at least maintained without the need of re-learning the embeddings nor the machine learning models.
The utilization of knowledge modeling techniques by means of knowledge graphs to analyze and model the teaming interdependence relationships along the team and task structure addresses the challenges outlined for situation awareness and its reasoning within the model and about it. The introduced concept of abstract activities as a mechanism to model performer/supporter team patterns leads to new ways to describe teaming processes and activities as well as teaming events that trigger interactions.
Knowledge Graph Modularization
Advances in digital and manufacturing technologies are strongly driven by data as a key enabler, creating opportunities beyond classic monitoring and predictive applications. In this context, KGs have strong application potential to create an integrated multi-perspective machine data space from heterogeneous data silos, lift and contextualize machine generated data, and facilitate cooperation between various domain experts and AI agents based on shared concepts. A key benefit of the proposed approach is that the resulting KGs support navigation across abstraction hierarchies, enabling bottom-up contextualization of raw data on the one hand, and top-down explanations by linking to lower levels of granularity on the other hand.
Dynamic Knowledge Graph Embeddings
Graph embeddings allow the use of graph-structured data for applications that, by definition, rely on numerical feature vectors as inputs. In TEAMING.AI the transformation of KGs into sets of numerical feature vectors is performed by embedding algorithms, which map the elements of the graph into a low-dimensional embedding space. However, these methods mostly assume a static knowledge graph, so subsequent updates inevitably require a re-run of the embedding process. In Teaming.AI we developed the NaviPy Approach which aims to maintain advantages of established embedding methods while making them accessible to dynamic domains. Relational Graph Convolutional Networks are adapted for efficient reconstructing node embeddings based solely on local neighborhoods. Preliminary results suggest that the performance of successive machine learning tasks is at least maintained without the need of re-learning the embeddings nor the machine learning models.