Knowledge about software evaluation,Development of recommendation for developing an information management software system
Melvil is a state-of-the-art information access tool. It uses innovative methods and technologies like ontology-based search, 3D representation of query and result, platform-independency through implementation by Java and full Web implementation. Melvil had been developed to a prototype stage and was partly beta-tested. In the project, the prototype was evaluated to see if the presumptions of the prior development were correct. The system was adapted to the need of a professional users (the pilot customer Austrian Chamber of Labour), installed at the organisation, partly introduced to their knowledge management processes and evaluated based on this adaptation and introduction. The main improvements realized within the project period were usability of the interfaces, particularly of the 3D interfaces, comprehensibility, ontology handling, the quality of search results, precision and accuracy and multi-lingual search. Evaluation war carried out as a continuous feedback process during the project period. Feedback of the pilot customers, of the project staff involved, and of independent test users was collected and used for adaptation and customisation. Evaluation methods included feedback circles, online bug reports, online feedback forms and "paper-pencil" questionnaires.
While investigating common term utility functions to determine how well a term describes a given set of documents, we found that a measure based on explicit trade-off between recall and precision is competitive to seven common measures used in information retrieval. Furthermore, our proposed measure allows the user to specify the trade-off between precision (i.e. terms which retrieve few, but very relevant documents) and recall (i.e. terms which retrieve many, but less relevant documents) in a simple and intuitive manner. Recent experiments on confidential data by SORA have shown that on average about eight of the top ten terms determined via this measure were considered relevant according to domain experts. This result could be used to describe unknown document collections concisely by a small set relevant terms (e.g. to reconstruct meaningful descriptions), to suggest relevant terms to the designer of an ontology node in Melvil (to improve the quality of the created concepts), and may also have applications in the biomedical domain (e.g. mining DNA data).