Skip to main content

The Development and Evaluation of New Methods for Editing and Imputation

Objective

The project will evaluate and compare the current methods for editing and imputation to establish current best practice methods. In addition, new methods for editing and imputation based on neural networks, support vector machine, fuzzy logic methodology and robust statistical methods will be developed and compared with the current best practice methods. The evaluation of different methods will require, (a) the creation of common data sets with known type of errors to be used by all the methods, and (b) the establishment of sound statistical criteria for the objective evaluation of the methods. Based on our evaluation, recommendations for the use of different methods for editing and imputation for different kinds of data sets will be made. A CD ROM containing the algorithms for different selected methods will be produced and widely disseminated for use by the NSIs and other private and public sector organisations interested in editing and imputation.

Objectives:
1. To establish a standard collection of data sets for EUREDIT
2. To develop a methodological evaluation framework and develop evaluation criteria
3. To establish a baseline by evaluating currently used methods for data editing and imputation.
4. To develop and evaluate a selected range of new techniques for data editing and imputation.
5. To evaluate different methods for edit and imputation and establish best methods for different types of data.
6. To disseminate the best methods via a single package for wider dissemination, and in a conference proceedings.

Work description:
In order to evaluate the editing and imputation methods, a set of representative data sets arising in social sciences (household surveys, business surveys, censuses, panel data) with known types of errors will be produced. The criteria for the evaluation of the methods in terms of Fellegi-Holt (1976) principles and operational efficiency will be established and agreed among the participants. Based on a review of currently used methods, a selection will be evaluated. This will establish the current best practice methods and also provide benchmark for the later phase of the project. Alongside the investigation of traditional methods, new methods for editing and imputation based upon advanced statistical and information technology techniques will be developed. Specifically, methods based upon: outlier robust methods and non-parametric regression, MLP neural networks, Radial Basis Function (RBF) neural networks, Correlation Matrix Memory (CMM) neural networks, Self-Organising Map (SOM) neural networks and Support Vector Machines (SVM), will be developed. This will involve the establishment of appropriate methodology, development of algorithms and application of methods to the selected data sets. All the methods (new and old) will be comparatively evaluated. This will form the basis for detailed recommendations about the optimal choice of methods in a wide range of common situations. The "best methods" will be selected for wider dissemination. This will be achieved through the development of portable software for the selected (best) methods. The CD containing the software will be produced and made available to the organisations interested in editing and imputation.

Milestones:
1. Selection and compilation of datasets for evaluating methods
2. Determination objective quality criteria for evaluating methodsL%3. Development and testing of selected new methods for error localisation
4. Development and testing of selected new methods for imputation
5. Evaluation of all (new and old) editing and imputation methods
6. Integration of the individual edit and imputation methods into a single

Funding Scheme

CSC - Cost-sharing contracts

Coordinator

OFFICE OF NATIONAL STATISTICS
Address
1, Drummond Gate
SW1V 2QQ London
United Kingdom

Participants (12)

CENTRAAL BUREAU VOOR DE STATISTIEK
Netherlands
Address
Prinses Beatrixlaan 428
2270 JM Voorburg
CREDIT SUISSE FINANCIAL PLANNING SOLUTIONS GMBH
Germany
Address
Wilhelm-theodor-roemheld-strasse 18
55130 Mainz
ISTITUTO NAZIONALE DI STATISTICA
Italy
Address
Via Cesare Balbo 16
00184 Roma
JYVAESKYLAEN YLIOPISTO
Finland
Address
Seminaarinkatu 15
40100 Jyvaskyla
QANTARIS GMBH
Germany
Address
Bahnhofstrasse 7
61476 Kronberg
ROYAL HOLLOWAY AND BEDFORD NEW COLLEGE
United Kingdom
Address
Egham Hill
TW20 0EX Egham, Surrey
STATISTICS DENMARK
Denmark
Address
Sejroegade 11
2100 Copenhagen
STATISTICS FINLAND
Finland
Address
Tyopajakatu 13
00022 Helsinki
SWISS FEDERAL STATISTICAL OFFICE
Switzerland
Address
Espace De L'europe 10
2010 Neuchatel
THE NUMERICAL ALGORITHMS GROUP LIMITED
United Kingdom
Address
Wilkinson House, Jordan Hill Road
OX2 8DR Oxford
UNIVERSITY OF SOUTHAMPTON
United Kingdom
Address
Highfield
SO17 1BJ Southampton
UNIVERSITY OF YORK
United Kingdom
Address
Heslington Hall
YO10 5DD York