Periodic Reporting for period 1 - KG-PROVENANCE (Tracing knowledge graph provenance from textual knowledge sources)
Período documentado: 2024-04-01 hasta 2026-03-31
Resumen del contexto y de los objetivos generales del proyecto
Knowledge graphs organise information as structured links between entities and are increasingly used in search, data integration, scientific discovery and decision-support systems. Much of the knowledge they contain originates in rapidly changing textual sources, such as scientific publications, online encyclopaedias and news. However, it is often difficult to trace a statement in a knowledge graph back to the text that supports it. This limits transparency, makes errors and outdated information harder to detect, and reduces trust in systems that rely on these resources. The challenge is particularly important in knowledge-intensive and high-risk areas such as biomedicine and pharmacovigilance, where users need to understand and verify the evidence behind information.
KG-PROVENANCE addressed this problem by studying how evolving textual knowledge can be connected to corresponding knowledge in knowledge graphs at scale. Its overall objective was to develop methods that identify the textual provenance of knowledge graph statements, detect differences between new textual evidence and existing graph content, and support appropriate updates with explicit evidence.
The project pursued three connected objectives. First, it aimed to create datasets linking changes in knowledge graphs to the evolving text that supports them. Second, it aimed to develop scalable methods for finding the most relevant supporting passages among very large and frequently changing text collections. Third, it aimed to develop provenance-backed methods for updating knowledge graphs when new evidence appears, including adding new knowledge, representing newly emerging entities and revising outdated statements.
The project responds to European priorities for transparent, trustworthy and human-centred artificial intelligence. Providing the evidence behind knowledge graph statements can improve traceability, facilitate human verification and help users understand why information is stored or why a change is proposed. This is especially relevant as knowledge graphs and large language models are increasingly combined in AI systems used in research and other knowledge-intensive settings.
The pathway to impact progresses from research datasets to scalable computational methods, open software and application-oriented validation. Public datasets and code allow researchers and developers to reproduce, evaluate and extend the work. Collaboration across knowledge graphs, natural language processing, machine learning and biomedical informatics connects the methods to realistic needs, including the integration of newly published biomedical evidence into existing knowledge resources. In the longer term, the project can contribute to more transparent and explainable AI systems by enabling users to inspect the evidence behind existing or newly proposed knowledge. More selective processing of relevant textual evidence can also reduce the computational and manual effort required to maintain large knowledge resources, supporting more timely and sustainable use of rapidly evolving information.
KG-PROVENANCE addressed this problem by studying how evolving textual knowledge can be connected to corresponding knowledge in knowledge graphs at scale. Its overall objective was to develop methods that identify the textual provenance of knowledge graph statements, detect differences between new textual evidence and existing graph content, and support appropriate updates with explicit evidence.
The project pursued three connected objectives. First, it aimed to create datasets linking changes in knowledge graphs to the evolving text that supports them. Second, it aimed to develop scalable methods for finding the most relevant supporting passages among very large and frequently changing text collections. Third, it aimed to develop provenance-backed methods for updating knowledge graphs when new evidence appears, including adding new knowledge, representing newly emerging entities and revising outdated statements.
The project responds to European priorities for transparent, trustworthy and human-centred artificial intelligence. Providing the evidence behind knowledge graph statements can improve traceability, facilitate human verification and help users understand why information is stored or why a change is proposed. This is especially relevant as knowledge graphs and large language models are increasingly combined in AI systems used in research and other knowledge-intensive settings.
The pathway to impact progresses from research datasets to scalable computational methods, open software and application-oriented validation. Public datasets and code allow researchers and developers to reproduce, evaluate and extend the work. Collaboration across knowledge graphs, natural language processing, machine learning and biomedical informatics connects the methods to realistic needs, including the integration of newly published biomedical evidence into existing knowledge resources. In the longer term, the project can contribute to more transparent and explainable AI systems by enabling users to inspect the evidence behind existing or newly proposed knowledge. More selective processing of relevant textual evidence can also reduce the computational and manual effort required to maintain large knowledge resources, supporting more timely and sustainable use of rapidly evolving information.
Trabajo realizado desde el comienzo del proyecto hasta el final del período abarcado por el informe y los principales resultados hasta la fecha
The project investigated how knowledge graphs can be maintained as the textual sources from which they draw information evolve. Historical changes in Wikipedia and Wikidata were analysed, and text-driven knowledge graph updating was defined as a new research problem. Five operations were formalised: confirming existing knowledge, adding relations between known entities, creating and connecting emerging entities, inferring links needed to integrate them into the graph, and deprecating outdated knowledge.
The main scientific achievement was EMERGE, a large-scale benchmark for evaluating knowledge graph updates driven by emerging text. A multi-stage pipeline was developed to align changes in Wikipedia passages with corresponding Wikidata changes, combining temporal alignment, large-language-model-based filtering and manual quality assessment. The dataset contains approximately 233,000 passages aligned with 1.45 million graph edits across seven yearly Wikidata snapshots from 2019 to 2025. Manual evaluation of 500 text–triple pairs found strong agreement with the automated annotations.
Information-extraction and large-language-model-based approaches were evaluated on EMERGE. The experiments showed that current systems can often extract plausible facts but struggle to convert them into correct operations on an existing graph. Key difficulties include alignment with existing entities and relations, integration of emerging entities, inference of connections not explicitly stated in the text, and identification of outdated knowledge. This demonstrated the need for methods that reason jointly over textual evidence and the current graph state.
Complementary work addressed related aspects of evolving knowledge. CYCLE introduced a graph-based contrastive learning method that uses relations added and removed over time to reduce temporal degradation in entity linking. It improved performance over the previous state of the art by 13.9% for a one-year time gap and 17.79% for a three-year gap. A further methodological line developed an architecture in which large language models interact with textual evidence and existing knowledge graphs to improve graph completion. It is being evaluated on five datasets, including four biomedical datasets. Additional work examined dynamic graph summarisation, modular entity disambiguation and multimodal graph enrichment through the conversion of images into entity-focused text.
Overall, the project produced new task definitions, datasets, computational pipelines and empirical evidence for building knowledge systems that can follow changing information while retaining explicit links between graph updates and their supporting evidence.
The main scientific achievement was EMERGE, a large-scale benchmark for evaluating knowledge graph updates driven by emerging text. A multi-stage pipeline was developed to align changes in Wikipedia passages with corresponding Wikidata changes, combining temporal alignment, large-language-model-based filtering and manual quality assessment. The dataset contains approximately 233,000 passages aligned with 1.45 million graph edits across seven yearly Wikidata snapshots from 2019 to 2025. Manual evaluation of 500 text–triple pairs found strong agreement with the automated annotations.
Information-extraction and large-language-model-based approaches were evaluated on EMERGE. The experiments showed that current systems can often extract plausible facts but struggle to convert them into correct operations on an existing graph. Key difficulties include alignment with existing entities and relations, integration of emerging entities, inference of connections not explicitly stated in the text, and identification of outdated knowledge. This demonstrated the need for methods that reason jointly over textual evidence and the current graph state.
Complementary work addressed related aspects of evolving knowledge. CYCLE introduced a graph-based contrastive learning method that uses relations added and removed over time to reduce temporal degradation in entity linking. It improved performance over the previous state of the art by 13.9% for a one-year time gap and 17.79% for a three-year gap. A further methodological line developed an architecture in which large language models interact with textual evidence and existing knowledge graphs to improve graph completion. It is being evaluated on five datasets, including four biomedical datasets. Additional work examined dynamic graph summarisation, modular entity disambiguation and multimodal graph enrichment through the conversion of images into entity-focused text.
Overall, the project produced new task definitions, datasets, computational pipelines and empirical evidence for building knowledge systems that can follow changing information while retaining explicit links between graph updates and their supporting evidence.
Avances que van más allá del estado de la técnica e impacto potencial esperado (incluida la repercusión socioeconómica y las implicaciones sociales más amplias del proyecto hasta la fecha)
KG-PROVENANCE produced results beyond the state of the art by shifting the focus from extracting isolated facts from text to updating a specific, existing knowledge graph. Conventional information-extraction systems generally identify entities and relations without considering what is already present in the graph, whether an entity is new, or whether an existing statement has become outdated. The project formalised text-driven knowledge graph updating as five operations: confirming existing knowledge, adding a relation between known entities, creating and connecting an emerging entity, inferring links required to integrate that entity into the graph, and deprecating outdated knowledge.
The central result is EMERGE, a benchmark that connects evolving Wikipedia passages with corresponding Wikidata updates over time. Its construction pipeline combines deterministic comparison of successive knowledge graph snapshots, temporal alignment with changes in text, large-language-model-assisted filtering, and manual validation. Unlike static information-extraction datasets, EMERGE evaluates whether a system produces the correct operation for the state of the graph at a particular time. It also distinguishes semantically plausible textual predictions from executable updates expressed using the graph’s canonical entities and relations.
A targeted knowledge-graph-aware experiment provided an important result beyond the original evaluation. The same large language model was tested with no graph context, with realistically retrieved graph context, and with the correct graph context supplied directly. Providing the correct context substantially improved near-exact and executable updating. Exact executable accuracy reached 50.3% for confirming existing knowledge, 39.6% for adding knowledge and 36.9% for deprecating outdated knowledge. However, performance fell considerably when the required context had to be retrieved automatically. Adding irrelevant retrieved information could also reduce performance, even when the correct context was present. Furthermore, the model remained far from perfect when given the correct entities, relations and existing statements.
These findings show that the difficulty of updating knowledge graphs is not simply a matter of converting names into graph identifiers. Two separate research challenges remain: retrieving and canonicalising the relevant graph context, and reasoning jointly over that context and the textual evidence to produce the correct update. The benchmark therefore provides both an evaluation resource and a diagnostic framework for developing future knowledge-graph-aware systems.
The results can contribute to more transparent and evidence-based maintenance of knowledge resources. Linking updates to supporting text can help researchers, developers and domain experts inspect why information is present, identify outdated statements and validate proposed changes. This is particularly relevant in rapidly evolving fields such as biomedicine, where newly published evidence must be integrated into existing resources while retaining human oversight. Publicly available datasets, code and evaluation tools support reuse, comparison and extension by other research groups.
Further uptake will require dedicated architectures that jointly retrieve, reason over and update knowledge graphs. Additional research is also needed to include dates, quantities and other literal attributes; extend the approach to languages beyond English and to specialised knowledge graphs; and address longer delays between changes in textual and structured sources. Demonstration in operational biomedical and industrial settings will be necessary to assess reliability, integration costs and the role of expert verification. Continued maintenance of provider-independent evaluation tools, reproducible local baselines and interoperable representations of textual evidence will also support long-term uptake. The principal current exploitation route is open and non-commercial, while commercial adoption would require application-specific integration, validation and support.
The central result is EMERGE, a benchmark that connects evolving Wikipedia passages with corresponding Wikidata updates over time. Its construction pipeline combines deterministic comparison of successive knowledge graph snapshots, temporal alignment with changes in text, large-language-model-assisted filtering, and manual validation. Unlike static information-extraction datasets, EMERGE evaluates whether a system produces the correct operation for the state of the graph at a particular time. It also distinguishes semantically plausible textual predictions from executable updates expressed using the graph’s canonical entities and relations.
A targeted knowledge-graph-aware experiment provided an important result beyond the original evaluation. The same large language model was tested with no graph context, with realistically retrieved graph context, and with the correct graph context supplied directly. Providing the correct context substantially improved near-exact and executable updating. Exact executable accuracy reached 50.3% for confirming existing knowledge, 39.6% for adding knowledge and 36.9% for deprecating outdated knowledge. However, performance fell considerably when the required context had to be retrieved automatically. Adding irrelevant retrieved information could also reduce performance, even when the correct context was present. Furthermore, the model remained far from perfect when given the correct entities, relations and existing statements.
These findings show that the difficulty of updating knowledge graphs is not simply a matter of converting names into graph identifiers. Two separate research challenges remain: retrieving and canonicalising the relevant graph context, and reasoning jointly over that context and the textual evidence to produce the correct update. The benchmark therefore provides both an evaluation resource and a diagnostic framework for developing future knowledge-graph-aware systems.
The results can contribute to more transparent and evidence-based maintenance of knowledge resources. Linking updates to supporting text can help researchers, developers and domain experts inspect why information is present, identify outdated statements and validate proposed changes. This is particularly relevant in rapidly evolving fields such as biomedicine, where newly published evidence must be integrated into existing resources while retaining human oversight. Publicly available datasets, code and evaluation tools support reuse, comparison and extension by other research groups.
Further uptake will require dedicated architectures that jointly retrieve, reason over and update knowledge graphs. Additional research is also needed to include dates, quantities and other literal attributes; extend the approach to languages beyond English and to specialised knowledge graphs; and address longer delays between changes in textual and structured sources. Demonstration in operational biomedical and industrial settings will be necessary to assess reliability, integration costs and the role of expert verification. Continued maintenance of provider-independent evaluation tools, reproducible local baselines and interoperable representations of textual evidence will also support long-term uptake. The principal current exploitation route is open and non-commercial, while commercial adoption would require application-specific integration, validation and support.