The SIFT project aims to construct a demonstration intelligent help system for online computer software manuals based on two key ideas: the Vector Space Model of information retrieval on the one hand and the use of distributed patterns to capture the meaning of textual information on the other. The final prototype will accept a user's query in natural language concerning the software and return a list of pointers into the manual texts indicating where passages answering the query might be found. These will be arranged in descending order of relevance to the query, allowing the user to investigate the most promising parts of the text first.
The project will also serve to demonstrate the usefulness of distributed patterns in practical Natural Language Processing systems and their compatibility with existing work on lexical databases and robust lexicalistic parsing.
The combination of VSM retrieval mechanisms and distributed patterns has already been demonstrated in a working retrieval system. However, this was created largely by manual means. SIFT will provide the technology for generating such a system automatically
Approach and Methodology
The SIFT system will consist of two main components. The document processing component will analyse an SGML tagged computer manual and associate with its different sections, subsections and individual sentences distributed patterns capturing the meaning --in gist-- of those textual units. The interactive query processing component will accept a user's input query and produce as output an ordered list of pointers to text portions.
The key ideas in the project are the use of robust lexicalistic parsing, the assignment of semantic cases to syntactic constituents and the extraction of distributed representations from machine readable dictionaries.
Exploitation and Future Prospects
The problem of access to textual information is one faced by every organisation. SIFT will demonstrate a technology which can directly address this task. Moreover, the techniques to be used are intended to be applicable to a wide range of related text processing tasks and could be incorporated into other products such as stylistic checkers, summarisation engines and machine assisted translation tools.
It is also expected that SIFT will yield theoretical insight into the applicability of automated processes to natural language processing, e.g. how distributed representations can be used to capture both word and sentence meanings, how a large, robust and general purpose semantic lexicon can be constructed automatically, and whether robust lexicalistic partial parsing is possible.
Plans for the commercial exploitation of SIFT technology in a commercial product are being investigated.