Virtualisation and Multimodal Exploration of Heritage on Nazi Persecution

Project Information

MEMORISE

Grant agreement ID: 101061016

DOI

10.3030/101061016

EC signature date 3 June 2022

Start date 1 October 2022

End date 30 September 2026

Funded under

• Culture, creativity and inclusive society

Total cost

€ 3 975 427,11

EU contribution

€ 3 975 427,11

Coordinated by

SYDDANSK UNIVERSITET
Denmark

Periodic Reporting for period 2 - MEMORISE (Virtualisation and Multimodal Exploration of Heritage on Nazi Persecution)

Reporting period: 2023-10-01 to 2024-09-30

Summary of the context and overall objectives of the project

With the passage of time and the accompanying disappearance of eyewitnesses, it becomes increasingly important to develop new forms of communicating and transmitting the memory and legacy of the Holocaust. Heritage related to Nazi Persecution (HNP) appears in various forms. Memories of eyewitnesses have been preserved in diaries, letters and testimonies (written or recorded), and registers (e.g. death records and deportation statistics) and photographs. Huge quantities of these resources exist across Europe but are not accessible to the general public. MEMORISE will use digital technology and AI methods to make Heritage related to HNP persistently available to all. MEMORISE will conserve, digitize and present these memories in a novel, interactive, and sustainable way. The preserved material will then be anchored to the places where the Nazi crimes happened. The project will create contemporary 3D models of concentration camps memorial sites as well generating 3D reconstructions of long-destroyed war-time buildings to be used for ‘virtual’ visits. Visitors either onsite, or with remote access will be able to undertake a novel multidimensional interactive AR experience with the content individualized for each visitor.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

We will develop:
- An infrastructure that assists 50 consortial and associated memorial sites and archives in processing 80,000+ HNP content items to make them persistently accessible to the general public.
-A suite of digital tools for presenting, narrating and engaging with HNP in new ways.
-Semi-automated, AI-based services in the form of 1, a data processing pipeline to process Dutch, English and German HNP materials, and 2, an AI engine to recommend individualised content to general public users.
-An intersectional toolkit – including tools and best-practice documentation for data processing, knowledge graph creation, curation, 3D modelling, narration and interactive visual exploration – to accelerate the use of digital tools in cultural heritage and adjacent cultural industries.
-An educational concept that evaluates the benefits and limitations of user interfaces for learning on and engaging with HNP in on-site and online settings, with a particular focus on the necessity of virtual solutions in times of a pandemic.
- A comprehensive evaluation methodology that stretches across the entire project duration to assess the real needs in handling HNP to reach a global audience.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

-MEMORISE will extend the use of linked data and knowledge graphs in the heritage domain by refining existing and defining new vocabularies to model HNP and by integrating these into NLP workflows to extract structured information from HNP resources. We will develop APIs for existing tools that support exploiting HNP vocabularies, and for constructing, editing and querying the resulting knowledge graphs. We will develop solutions to integrate large-size data like videos and 3D models with a data repository that provides a user-friendly interface that supports curation processes. One particular outcome will be a translation of the Dutch WW2 thesaurus for international adoption.
-The digitised collections (metadata and historical documents) of the partners within MEMORISE, and in the future institutions outside of the project, will be not only connected with each other, but also made available for virtualization, education, research, museum use and communication through a rich annotation and knowledge extraction processes aided by AI powered NLP methods and curated by human experts that will substantially improve efficiency and effectiveness of the process. The approach will be easily scalable to related cultural heritage domains.
-MEMORISE will advance the implementation and handling of 3D models by deploying and extending best practices in a novel pipeline model, focusing on sustainable interoperability within the context of the semantic web and sensitive heritage. The specific novelty will be the context-specific integration of knowledge extracted from source materials into spatiotemporal representations of HNP memorial sites. For that purpose, geospatial and temporal metadata will be annotated and connected to a geospatial ontology, which will be tailored for individual memorial sites and thus provide a model to adopt by other historical sites.
-MEMORISE will propose a novel synthetic agent (AI engine) grounded on neuroscientific principles to under- stand the users’ cognitive state and assist in the exploration of historical datasets. This agent will be based on a cognitive architecture theory that implements its perception, cognition and action capabilities. The synthetic agent will be able to dynamically build an HNP user model based on multimodal signal interpretation to assess their cognitive states. It will learn from the interactions with the current user and use information from previous experiences, forming a memory unit that the agent will use to refine the content proposal and create an individual experience.
-MEMORISE will deploy a HNP platform to assist researchers for the first time getting an overview of existing materials. Geo-temporal visualizations will arrange those materials in an appropriate historical context, and a novel map-based network will make the Nazi transportation infrastructure visible. We will develop a comparative reader to allow the parallel reading of Nazi persecution experiences, and we will further explore new ways of storytelling enabled for the first time by AI-based services tailored to the users’ own off-site and on-site experiences. The presentation will include transnational data to highlight global relations across HNP resources.
-MEMORISE for the first time offers a holistic integration of digital heritage culture, considering ethical, educational and social media guidelines and a comprehensive multiphase implementation testing. It will be the fir in-depth study on using digital technologies for engaging with HNP and difficult pasts. It will involve many users to evaluate the educational capabilities of our solutions.

MEMORISE Logo