Skip to main content

Index based Statistical Analysis of Large Text Corpora

Objective

The statistical analysis of large text corpora is a fundamental method for gaining insights into the structure of language, e.g. for grammar development, machine translation, terminology and named entity extraction, text correction, semantic text analysis, and others. Progress in these fields helps to improve related applications in information science (search engine technology) and many other text oriented disciplines.
The core contribution of this project is a new methodology aimed at fundamentally improving statistical analysis of large text corpora. A weakness of current methods in corpus analysis is insufficient use of contextual information. Properly understanding the role, function and meaning of a phrase or word (which is important for many applications, e.g. for translation, search, etc.) is often only possible when taking sentence/paragraph contexts into account. We want to develop and study a new representation of corpora which is superior to present formats in three respects. Most importantly, it offers a much better use of contextual information. At the same time it helps to better distinguish between arbitrary and meaningful parts of text and gives hints on how to compose/decompose phrases. With these properties, the new representation gives a basis for fundamentally improving statistical analysis of corpora. The new representation is derived from a special text index structure which gives immediate access to contexts of any size. The index imposes a natural graph structure on the the phrases in the corpus, which implies that interesting graph-based statistical methods can be applied. Further more it can be efficiently constructed and updated in practice.
To practically demonstrate the large potential of the new methodology in NLP we will concentrate on the machine translation where we expect to achieve improved translation methods for words and phrases.

Call for proposal

FP7-PEOPLE-2013-IEF
See other projects for this call

Coordinator

LUDWIG-MAXIMILIANS-UNIVERSITAET MUENCHEN
Address
Geschwister Scholl Platz 1
80539 Muenchen
Germany
Activity type
Higher or Secondary Education Establishments
EU contribution
€ 161 968,80
Administrative Contact
Klaus Schulz (Prof.)