Periodic Reporting for period 2 - MUHAI (Meaning and Understanding in Human-centric AI)
Período documentado: 2022-04-01 hasta 2023-09-30
The MUHAI project explores a cognitively motivated approach to push the envelope of human-centric AI technology to come to grips with meaning and understanding. In MUHAI understanding is the process of constructing meaning by casting episodic events into coherent narratives that provide declarative answers to specific questions and thereby explain experiences and integrate them into a personal dynamic memory. The implemented process of understanding uses information from both syntactic and semantic interpretation and from inferences based on the personal dynamic memory, in order to fill in unexpressed or un-observable information, e.g. via logical reasoning and mental simulation. For this, the MUHAI project sets out to perform ambitious research and implementation efforts.
The MUHAI project has adhered to the principles of understanding-by-building and iterative development, which allows components to be operational and testable from the start, and which informs the next iterations of development. Moreover, the implementation activities were able to draw from the cognitive foundations that the partners collectively established in the beginning of the project. The software environment CANVAS has been implemented, focusing currently on the everyday activity use case. CANVAS takes textual instructions, such as recipes, as input. CANVAS is not based on a pipeline of processing components, but features an ensemble of modules that answer narrative questions that arise in the understanding process in a concerted manner. The MUHAI approach to understanding is also ported to several domains from the humanities, namely the social sciences, history and economics. Here, knowledge-based technologies -- using collections of knowledge graphs -- are employed to construct and display narratives of the individual domains at hand.
The MUHAI project has adhered to the principles of understanding-by-building and iterative development, which allows components to be operational and testable from the start, and which informs the next iterations of development. Moreover, the implementation activities were able to draw from the cognitive foundations that the partners collectively established in the beginning of the project. The software environment CANVAS has been implemented, focusing currently on the everyday activity use case. CANVAS takes textual instructions, such as recipes, as input. CANVAS is not based on a pipeline of processing components, but features an ensemble of modules that answer narrative questions that arise in the understanding process in a concerted manner. The MUHAI approach to understanding is also ported to several domains from the humanities, namely the social sciences, history and economics. Here, knowledge-based technologies -- using collections of knowledge graphs -- are employed to construct and display narratives of the individual domains at hand.
The central contributions of the work performed in MUHAI concern the primary objectives of artificial intelligence that is human-centric instead of being solely data-centric. In particular understanding is at the heart of the MUHAI research. The MUHAI domain of understanding everyday activities can primarily impact household robotics and domestic as well as restaurant assistants. For this an understanding system for instructions concerning everyday activities has been implemented in which two components (one for common sense and encyclopedic knowledge and one for the dynamic situation model) comprise the personal dynamic memory (PDM). A physics-based mental simulation of the execution is triggered by the instructions queries and updates the PDM. The ILR parser analyses the textual instructions, reads in the situation model at the onset and constantly grounds and resolves linguistic forms by queries to the simulation, ontology and situation model. As a consequence of the pandemic situation, new challenges for the food industry emerged. Specifically, the need to forecast demand for orders of dishes has become an additional focal part of the application developed in the project. Here, concrete impacts, e.g. by reducing food waste, can be seen as a result of a better understanding of this domain provided by MUHAI technologies.
The consortium is preparing various challenges and benchmarks for individual application areas. These benchmarks will be refined as the targets and technology of the project to enter the next iteration phase. Furthermore, strong progress has been made at formalising cognitive constructs related to the development of personal dynamic memory as a data structure that can be employed for inferences and learning. All partners have been making significant progress for building the generic components to build AI systems that understand, and for demonstrating their effectiveness in the chosen application domains. Significant steps have been made by introducing APIs for various components so that they can be used in a RESTful architecture (and therefore integrated in platforms such as AI4EU) and grouped in the CANVAS library. Concrete achievements have been made for components that perform language processing (including scaling and semantic frame extraction), mental simulations by physical simulation, and linking these simulations to semantic descriptions.
The project's first results towards a truly introspective system lies in the design of the cognitive architecture of the CANVAS system. As opposed to traditional pipeline configurations, different components are able to communicate with each other in a more distributed process. The main innovation concerning introspection therefore lies in the integration of technologies, that, by themselves, have existed prior, but were not used as an ensemble for self-reflective understanding. A clear illustration is the cooking domain. In order to understand a recipe, the analysis of natural language must constantly interface with corresponding simulations in a virtual representation of the world. For example, by revisiting a recipe a second time after the system has learned a first set of parameters, the CANVAS system can change the order of the individual tasks contained in the instructions, avoid unnecessary waiting times by reflecting on temporal information from prior simulations, and so on.
The need for strong explicability is an additional tenet of human-centric AI. AI needs to be able to explain its reasoning, and explain why certain decisions were made or actions were taken. This is also one of the things with which current language understanding methods based on subsymbolic AI struggle the most. In contrast, the methods developed in the MUHAI project make it possible to explain the system’s understanding process. This includes for example deep natural language understanding based on constructional language processing, the utilization of ontological knowledge and reasoning including memories of previous experiences, self-reflection and introspection, as well as narratives constructed from knowledge graphs.
Significant exploratory work has been done to identify which approach should be used to embed ethical considerations in human-centric AI and to enhance the trust-worthiness of AI systems. The project has argued for the need for increasing understanding in AI systems. For the cooking domain, the ethical issues are quite different compared to the social domain, in which the social stance of the producer or consumer of information is unavoidably ethically colored. This objective also includes care about privacy and security issues which will be tackled by storing PDMs locally with the user who owns the data and knowledge, without the need for centralized data management and hence the risk of data theft. At this point, this aspect has not been tackled but will become part of the cognitive architecture that is currently being developed.
The project's first results towards a truly introspective system lies in the design of the cognitive architecture of the CANVAS system. As opposed to traditional pipeline configurations, different components are able to communicate with each other in a more distributed process. The main innovation concerning introspection therefore lies in the integration of technologies, that, by themselves, have existed prior, but were not used as an ensemble for self-reflective understanding. A clear illustration is the cooking domain. In order to understand a recipe, the analysis of natural language must constantly interface with corresponding simulations in a virtual representation of the world. For example, by revisiting a recipe a second time after the system has learned a first set of parameters, the CANVAS system can change the order of the individual tasks contained in the instructions, avoid unnecessary waiting times by reflecting on temporal information from prior simulations, and so on.
The need for strong explicability is an additional tenet of human-centric AI. AI needs to be able to explain its reasoning, and explain why certain decisions were made or actions were taken. This is also one of the things with which current language understanding methods based on subsymbolic AI struggle the most. In contrast, the methods developed in the MUHAI project make it possible to explain the system’s understanding process. This includes for example deep natural language understanding based on constructional language processing, the utilization of ontological knowledge and reasoning including memories of previous experiences, self-reflection and introspection, as well as narratives constructed from knowledge graphs.
Significant exploratory work has been done to identify which approach should be used to embed ethical considerations in human-centric AI and to enhance the trust-worthiness of AI systems. The project has argued for the need for increasing understanding in AI systems. For the cooking domain, the ethical issues are quite different compared to the social domain, in which the social stance of the producer or consumer of information is unavoidably ethically colored. This objective also includes care about privacy and security issues which will be tackled by storing PDMs locally with the user who owns the data and knowledge, without the need for centralized data management and hence the risk of data theft. At this point, this aspect has not been tackled but will become part of the cognitive architecture that is currently being developed.