Servizio Comunitario di Informazione in materia di Ricerca e Sviluppo - CORDIS

Guidelines and a methodology for the construction of diagnostic data

The project involves defining guidelines and a methodology for the construction of diagnostic data (test suites) and designing and implementing related tools. The guidelines will be validated by constructing application-specific diagnostic data for French, German and English and testing the data on a number of applications ranging from parsers through grammar checkers to controlled language checkers. In addition to devising guidelines, the project investigates techniques and design and implement tools that will facilitate the construction, use and manipulation of test suites, such as a database for storing and manipulating test data, and tools for the (semi)automatic generation of test suites. An advantage of automation is that this allows dynamic test suite construction, where test data can be replaced by new data which tests the same phenomena. In this way, it may be possible to overcome one of the problems that sometimes crops up in system evaluation, namely that developers can tune their application so that it deals with static test data.

The project has designed a methodology for the construction of test suites and produced a database of parallel test suites in three languages of some 4000 annotated items per language, with particular reference to three application types: parsers, grammar checkers and controlled language checkers. The test material is relevant to any developer or user of systems with grammar components who wish to test, benchmark or evaluate those systems. The test suites are fully documented and mounted on a database for ease of access and manipulation. Particular attention has been paid to the design and implementation of an efficient annotation scheme.

Reported by

University of Essex
Wivenhoe Park
CO4 3SQ Colchester
United Kingdom
See on map