Periodic Reporting for period 3 - MUHAI (Meaning and Understanding in Human-centric AI)
Reporting period: 2023-10-01 to 2025-03-31
The MUHAI project explored a cognitively motivated approach to push the envelope of human-centric AI technology to come to grips with meaning and understanding. In the MUHAI approach understanding is:
“The process of constructing meaning by casting episodic events into coherent narratives that provide declarative answers to specific questions and thereby explain experiences and integrate them into a personal dynamic memory.”
The process of understanding that was implemented in the MUHAI project uses information from both syntactic and semantic interpretation and from inferences based on the personal dynamic memory, in order to fill in unexpressed or un-observable information, e.g. via logical reasoning and mental simulation. For this, the MUHAI project performed ambitious research and implementation efforts.
The MUHAI project has adhered to the principles of understanding-by-building and iterative development, which allowed components to be operational and testable from the start, and which informs the next iterations of development. Moreover, the implementation activities were able to draw from the cognitive foundations that the partners collectively established in the beginning of the project. The final software library, i.e. CANVAS, has been implemented, focusing on and serving both use cases. CANVAS is not based on a pipeline of processing components, but features an ensemble of modules that answer narrative questions that arise in the understanding process in a concerted manner. The MUHAI approach to understanding is also ported to several domains from the humanities, namely everyday activities, the social sciences, history and economics. In all cases, knowledge-based technologies -- using collections of knowledge graphs -- are employed to construct and display narratives of the individual domains at hand. The MUHAI project has, therefore, successfully concluded all necessary actions that open a pathway towards a human-centric AI approach to tackle meaning and understanding in a computational manner.
“The process of constructing meaning by casting episodic events into coherent narratives that provide declarative answers to specific questions and thereby explain experiences and integrate them into a personal dynamic memory.”
The process of understanding that was implemented in the MUHAI project uses information from both syntactic and semantic interpretation and from inferences based on the personal dynamic memory, in order to fill in unexpressed or un-observable information, e.g. via logical reasoning and mental simulation. For this, the MUHAI project performed ambitious research and implementation efforts.
The MUHAI project has adhered to the principles of understanding-by-building and iterative development, which allowed components to be operational and testable from the start, and which informs the next iterations of development. Moreover, the implementation activities were able to draw from the cognitive foundations that the partners collectively established in the beginning of the project. The final software library, i.e. CANVAS, has been implemented, focusing on and serving both use cases. CANVAS is not based on a pipeline of processing components, but features an ensemble of modules that answer narrative questions that arise in the understanding process in a concerted manner. The MUHAI approach to understanding is also ported to several domains from the humanities, namely everyday activities, the social sciences, history and economics. In all cases, knowledge-based technologies -- using collections of knowledge graphs -- are employed to construct and display narratives of the individual domains at hand. The MUHAI project has, therefore, successfully concluded all necessary actions that open a pathway towards a human-centric AI approach to tackle meaning and understanding in a computational manner.
The central contributions of the work performed in MUHAI concern the primary objectives of artificial intelligence that is human-centric instead of being solely data-centric. In particular understanding is at the heart of the MUHAI research. The MUHAI domain of understanding everyday activities can primarily impact household robotics and domestic as well as restaurant assistants. For this an understanding system for instructions concerning everyday activities has been implemented in which two components (one for common sense and encyclopedic knowledge and one for the dynamic situation model) comprise the personal dynamic memory (PDM). A physics-based mental simulation of the execution is triggered by the instructions queries and updates the PDM. The Incremental Recruitment Language parser analyses the textual instructions, reads in the situation model at the onset and constantly grounds and resolves linguistic forms by queries to the simulation, ontology and situation model. As a consequence of the pandemic situation, new challenges for the food industry emerged. Specifically, the need to forecast demand for orders of dishes has become an additional focal part of the application developed in the project. Here, concrete impacts, e.g. by reducing food waste, can be seen as a result of a better understanding of this domain provided by MUHAI technologies. The project results have not only been exhibited in numerous AI roadshows and conferences but have also lead to spin-off projects that take the results to the next level of technological readiness. ain provided by MUHAI technologies.
The consortium prepared and published various challenges and benchmarks for individual application areas. Furthermore, strong progress has been made at formalising cognitive constructs related to the development of personal dynamic memory as a data structure that can be employed for inferences and learning. All partners have been making significant progress for building the generic components to build AI systems that understand, and for demonstrating their effectiveness in the chosen application domains. Significant steps have been made by introducing APIs for various components so that they can be used in a RESTful architecture and grouped in the CANVAS library. Concrete achievements have been made for components that perform language processing (including scaling and semantic frame extraction), mental simulations by physical simulation, and linking these simulations to semantic descriptions.
The project's results towards a truly introspective system is manifested in the design of the cognitive architecture of the CANVAS system. As opposed to traditional pipeline configurations, different components are able to communicate with each other in a more distributed process. The main innovation concerning introspection therefore lies in the integration of technologies, that, by themselves, have existed prior, but were not used as an ensemble for self-reflective understanding. A clear illustration is the cooking domain. In order to understand a recipe, the analysis of natural language must constantly interface with corresponding simulations in a virtual representation of the world. For example, by revisiting a recipe a second time after the system has learned a first set of parameters, the CANVAS system can change the order of the individual tasks contained in the instructions, avoid unnecessary waiting times by reflecting on temporal information from prior simulations, and so on.
The need for strong explicability is an additional tenet of human-centric AI. AI needs to be able to explain its reasoning, and explain why certain decisions were made or actions were taken. This is also one of the things with which current language understanding methods based on subsymbolic AI struggle the most. In contrast, the methods developed in the MUHAI project make it possible to explain the system’s understanding process. This includes for example deep natural language understanding based on constructional language processing, the utilization of ontological knowledge and reasoning including memories of previous experiences, self-reflection and introspection, as well as narratives constructed from knowledge graphs.
Significant work has been done to identify which approach should be used to embed ethical and societal considerations in human-centric AI and to enhance the trust-worthiness of AI systems. The project has argued for the need for increasing understanding in AI systems. For the cooking domain, the ethical and societal issues are quite different compared to the social domain, in which the social stance of the producer or consumer of information is unavoidably ethically coloured. This objective also includes care about privacy and security issues which will be tackled by storing PDMs locally with the user who owns the data and knowledge, without the need for centralized data management and hence the risk of data theft.
The project's results towards a truly introspective system is manifested in the design of the cognitive architecture of the CANVAS system. As opposed to traditional pipeline configurations, different components are able to communicate with each other in a more distributed process. The main innovation concerning introspection therefore lies in the integration of technologies, that, by themselves, have existed prior, but were not used as an ensemble for self-reflective understanding. A clear illustration is the cooking domain. In order to understand a recipe, the analysis of natural language must constantly interface with corresponding simulations in a virtual representation of the world. For example, by revisiting a recipe a second time after the system has learned a first set of parameters, the CANVAS system can change the order of the individual tasks contained in the instructions, avoid unnecessary waiting times by reflecting on temporal information from prior simulations, and so on.
The need for strong explicability is an additional tenet of human-centric AI. AI needs to be able to explain its reasoning, and explain why certain decisions were made or actions were taken. This is also one of the things with which current language understanding methods based on subsymbolic AI struggle the most. In contrast, the methods developed in the MUHAI project make it possible to explain the system’s understanding process. This includes for example deep natural language understanding based on constructional language processing, the utilization of ontological knowledge and reasoning including memories of previous experiences, self-reflection and introspection, as well as narratives constructed from knowledge graphs.
Significant work has been done to identify which approach should be used to embed ethical and societal considerations in human-centric AI and to enhance the trust-worthiness of AI systems. The project has argued for the need for increasing understanding in AI systems. For the cooking domain, the ethical and societal issues are quite different compared to the social domain, in which the social stance of the producer or consumer of information is unavoidably ethically coloured. This objective also includes care about privacy and security issues which will be tackled by storing PDMs locally with the user who owns the data and knowledge, without the need for centralized data management and hence the risk of data theft.