From a technical perspective, we have worked diligently over the first 24 months to complete Work package 1 (WP1), i.e. constructing the main research corpus, converting available texts into the TEI-XML format and loading them into PhiloLogic, an open-source full-text search and analysis engine. Thanks to institutional agreements with the University of Chicago, Oxford, and the Bibliothèque nationale de France (BnF), we were able to federate existing digital collections into two main research corpora: ‘Canon’, which contains 3,500 transcribed texts with little or few errors; and ‘Archive’, which contains almost 10,000 texts that are the result of automated optical character recognition (OCR), and therefore highly variant in terms of noise and error-rates.
In Y2 we ran the Text-PAIR sequence aligner over the combined corpus to identify 18th-century text reuses. This process generated 2.5 million initial text alignments, which formed the basis of our initial experiments and led to multiple paper presentations and a published article in the journal Humanités Numériques. We also realised, somewhat surprisingly, that a large majority of these alignments could be classified as ‘noise’, e.g. repeated passages that occur in large quantities across texts but that cannot be considered a ‘reuse’ and therefore outside the bounds of our project. We thus set about conceiving of an automated method of filtering this noise from the alignments leveraging the power of deep-learning algorithms and language models to train a model to recognise paratextual noise in identified alignments (see below). Once the filter was deployed, we were able to eliminate almost 90% of the identified alignments as ‘noise’, leaving us with just over 250,000 pairwise reuses drawn from our main research corpus.
The team next developed a data model and structure for the above alignments, following semantic-web and open-science standards, which was implemented in PostgreSQL on the project server. A white paper was published on the project’s website that describes the data model (see below).
In Y3 we began work in two main areas: first, our new software engineer developed a web interface for annotating our filtered alignments collected in WP1 (see image 1). This interface allows team members to classify alignments as 'valid' 'invalid' or 'uncertain', as well as compile lists of annotated alignements for further model training and analysis. The resulting annotated data will
Work package 2 (WP2) is also well underway.we have begun analysing the sequence alignment data to model the most ‘influential’ authors and texts in the database. Moving forward, the we will evaluate 18th-century neural network language models trained using either context- independent word embeddings to generate conceptual maps and semantic tags for the alignments. WP3 will be repurposed to focus more intensely on the language-modelling and semantic analysis aspects while network analysis and alignment annotation will continue in WP2.
