Service Communautaire d'Information sur la Recherche et le Développement - CORDIS

TEMAA: a testbed study of evaluation methodologies-authoring aids

The project concerns the provision of a generally applicable framework for the evaluation of natural language processing (NLP) products. The project also studies project evaluation to some extent, but the focus is on adequacy evaluation. The framework is expressed as a number of guidelines to be followed when setting up an evaluation of a specific type of product. These guidelines are implemented in an object-oriented environment to facilitate modelling of user requirements and system characteristics. The result is the development of a parametrized test bed (PTB). Test materials for the testing of spelling checkers' lexical coverage have also been constructed. The TEMAA framework is modelled in CLIPS following an object-oriented programming (OOP) paradigm. The project has also developed an experimental tool supporting automatic spelling checker checking (ASCC), which is interfaced with CLIPS. The tool is a collection of programs in WINBATCH, C, and Perl. The results of the project will be documented in a set of reports and an evaluation package covering some of the test materials necessary for the evaluation of spelling checkers for Danish, English and Italian will be produced. A program, ASCC, which automatically transforms correctly spelled words into misspelled words, and measures the ability of the spelling checker to find errors and to produce a good suggestion will be developed and a prototype of a test modelling tool for the description of an evaluation of some NLP product will be provided.

Reported by

See on map