Wspólnotowy Serwis Informacyjny Badan i Rozwoju - CORDIS

Testing of Practical Evaluation Tools Framework through case studies

This work package sought to test PETUS Decision Support System (DSS) with public and private sector organisations in order to prove its effectiveness. The testing process developed into two main stages: a preliminary stage (development of a testing concept and strategy, before the official start of WP5) and a core testing stage with two sub-stages: a methodological (development of testing criteria, procedures and instruments); and an operational one (provision of structured feedback from end users through interviews and testing case-studies).

A pilot testing with a restricted focus on only one sector of urban infrastructure (Green/Blue) was carried out in two Bulgarian towns. The main issues traced were: general public awareness on and local authorities’ commitment to sustainable development, attitude towards urban public green spaces; evaluation tools currently applied and communication patterns between authorities and citizens; local business involvement into public greenery maintenance. The pilot testing results provided both support for further discussion on PETUS DSS focus and suggestions for the development of the testing process itself.

The testing models and instruments were developed in accordance with the initial testing strategy based on the three effectiveness criteria chosen (credibility – scientific/technical soundness; saliency – ability to address particular practical problems; legitimacy – acceptability by end users.). As, being reflexive to end users’ demands and preferences, PETUS took the form of a web-based interactive DSS, the initially discussed effectiveness testing criteria had to be complemented by a particular focus on its accessibility and usability. Accessibility regarded smooth transformation of information and regarded technical components and content management of the website (to provide information in a clear and simple manner and by an understandable mechanism to navigate within and between pages). Usability meant the degree of practical use and implementation of the tool measured by the satisfaction with the results of the tool implementation.

According to the peculiarities of the target groups, identified in the testing concept, three testing models were developed. Testing model 1 (TM1) comprised an individual filling-in of a Respondent Questionnaire (RQ) after a visit to PETUS website or a presentation by PETUS team member. The model was considered appropriate for graduate and PhD students, politicians, local businessmen and NGOs. Testing Model 2 (TM2) supposed face-to-face communication of an interviewer with an individual or a group of respondents. TM2a comprised an individual interview (open but structured discussion after a presentation of PETUS or a personal website visit). It was considered appropriate for contacting academic staff, technical experts and NGOs. An Interviewer’s Questionnaire (IQ) was to be filled in by the interviewer himself. TM2b was based on a Focus Group Discussion and was to identify the level of readiness for participation in PETUS further development and to provide innovative suggestions about the further development of the DSS content and functions. This model was considered appropriate for academic staff, technical experts and NGO members. Testing model 3 (TM3) was developed for testing the effectiveness of the PETUS DSS with local (technical) experts on real case studies. It was supposed to provide feedback on the PETUS DSS effectiveness through an experimental matrix application to ongoing or recently accomplished urban plans/ programmes.

It comprised four steps:
- review of the testing case study;
- filling in the PETUS matrix;
- structured interview with the end users;
- analysis of testing results.

The testing of PETUS (March – November 2005) comprised a number of interviews taken in three periods (March – May, June – August, September – November). TM1and TM2a inquiries were carried out with 84 respondents from all partner countries. Respondents from all the target groups of the project were reached: municipal experts and politicians, academic staff, NGO members. Feedback was provided on the estimation of PETUS usability, effectiveness and accessibility. Guidance needed to facilitate the DSS use was asked about. Answers to open questions provided comments about the contents and structure of PETUS and about technical aspects of the website-based DSS.

The PETUS matrix was also tested with end users in six real case studies from on-going urban development practice in 5 European countries - UK, France, Denmark, Austria and Bulgaria. This was the basis for identifying possible improvements, existing implementation barriers and future research challenges.

Powiązane informacje

Reported by

1, Chr. Smirnenski Blvd
1046 SOFIA
Śledź nas na: RSS Facebook Twitter YouTube Zarządzany przez Urząd Publikacji UE W górę