The REGINDEX project sets out to develop new efficient algorithms and data structures for searching large amounts of structured data. This objective is becoming a necessity in several scientific fields, such as bioinformatics, databases, and web engines, having in common the fact that the data they need to process is being produced at exponentially-increasing rates. As an example, consider the case of DNA sequencing. It took scientists 13 years (1990-2003) to complete the first draft of the human genome; this feat costed about 2.7 billion US dollars. By 2006, the cost of sequencing a Human genome dropped to about 14 million US dollars. Today, a next-generation sequencing machine can sequence a complete human genome in less than 30 hours for less than 1000 US dollars. This technological revolution has already led to the generation of databases containing hundreds of thousands of Human genomes (each consisting of about 3 billion nucleotides, i.e. "DNA letters") and is already revolutionizing medicine, paving the way to personalized (genome-based) treatments. From a computational point of view however, this revolution poses enormous algorithmic challenges. Just storing 1 million uncompressed Human genomes would require about 1000 Terabytes of available disk space, let alone pre-processing this data in order to support fast searches on it (a functionality that is vital in order to discover whether a particular DNA mutation has been seen before).
REGINDEX tackles this challenge by extending the concept of “sorting” to structured data. Sorting is a familiar concept that often allows searching data much faster: consider, for example, the sorted words in a dictionary. REGINDEX’s main goal is to show that this simple idea can be extended to much more structured data, even when the data is compressed. More in detail, the project focuses on indexing labeled graphs and regular languages for substring search queries. One can imagine a labeled graph as a generalization of a simple text. While in a text, letters occur consecutively, in a labeled graph one can specify which “jumps” between different portions of the text are allowed (and which are not). For example, a labeled graph can be used to encode a family of related genomes: a particular sub-sequence could be missing in a genome while it could be present in others. A related concept from theoretical computer science is that of “regular language”: a set of strings (e.g. 1000 Human genomes) can be encoded compactly as a set of rules (called “regular expressions”) specifying how to generate any string from the set. REGINDEX’s broad objective is to develop compressed representations for labeled graphs and regular languages, supporting efficient substring queries: to find out if and where a query short string (e.g. a short DNA sequence) appears as a substring in the indexed set of strings. In order to achieve this ambitious goal, REGINDEX introduces the novel concept of “co-lexicographic partial order”: a powerful tool that allows sorting (and therefore also compressing and indexing) labeled graphs and regular languages, despite their complex structure. Ultimately, the techniques developed within the REGINDEX project will make it possible to store millions of Human genomes in just a few Gigabytes, while at the same time supporting fast substring search queries on the compressed database.
