Final Report Summary - RISQ (Representativity indicators for survey quality)
Project RISQ (Representativity indicators for survey quality) started with three goals in mind; to develop and test quality indicators that
1) enable evaluation and comparison of the representativity of response to surveys and registers,
2) can be used as tools in monitoring response during survey data collection, and
3) can be used as objective functions in improving representativity of response through tailored survey designs.
At the start of the project basic ideas had been developed of what quality indicators may look like and how they may be estimated. However, the indicators were incomplete, and above all needed international comparison and empirical validation. The project structure and format were directed at extension and elaboration of existing theory and at international validation of indicators using datasets from five countries.
In order to be really informative and useful, indicators need to have a number of pre-specified properties. If the goal is to compare and evaluate response, then indicators must be easy to interpret, easy to compute and easy to translate to practical settings. If the goal is to monitor survey data collection, then indicators must be relevant and must allow for a top-down assessment of the impact of various population characteristics and survey design features. If the goal is to re-design survey data collection in order to improve representativity, then indicators must be effective.
With these goals and these requisites in mind, the project was divided in a theoretical part and an empirical part. The empirical part consisted of two designed data collection experiments.
As its major achievement RISQ has developed representativity indicators; simply termed as R-indicators and partial R-indicators. The indicators are documented, extensively tested and implemented as open source modules for R and SAS. The analyses, tests and experiments conducted within the project show that the indicators are valuable tools and fit for purpose. Survey designs tailored with the help of the indicators improved the representativity of response significantly. At workshops and conferences the project has advocated the use of the indicators as standards in quality reports. Besides in evaluating and designing survey data collection, the indicators also proved useful in monitoring the completion of registry data. The indicators are implemented in SAS and R open source modules. They can be downloaded from the website. Manual and test data sets are available.
Formally RISQ ended July 2010 with the release of software in SAS and R and the publication of theoretical and practical overview papers. Informal the project will be continued. In November 2010 release 2 of the software is planned that will add standard errors to partial R-indicators. In the spring of 2011 release 3 will include population-based R-indicators. The RISQ website, http://www.risq-project.eu will be updated on a regular basis and will contain the future releases. RISQ related questions and a request to receive future announcements can be send to risq@cbs.nl.
Contact persons for RISQ are:
- Statistics Netherlands: Barry Schouten (overall coordinator, bstn@cbs.nl) and Jelke Bethlehem (jbtm@cbs.nl)
- University of Southampton: Natalie Shlomo (N.Shlomo@soton.ac.uk) and Chris Skinner (C.J.Skinner@soton.ac.uk)
- Statistics Norway: Øyvin Kleven (Oyvin.Kleven@ssb.no)
- Katholieke Universiteit Leuven: Geert Loosveldt (Geert.Loosveldt@soc.kuleuven.be)
- Statistical Office of the Republic of Slovenia: Katja Rutar (Katja.Rutar@gov.si)
1) enable evaluation and comparison of the representativity of response to surveys and registers,
2) can be used as tools in monitoring response during survey data collection, and
3) can be used as objective functions in improving representativity of response through tailored survey designs.
At the start of the project basic ideas had been developed of what quality indicators may look like and how they may be estimated. However, the indicators were incomplete, and above all needed international comparison and empirical validation. The project structure and format were directed at extension and elaboration of existing theory and at international validation of indicators using datasets from five countries.
In order to be really informative and useful, indicators need to have a number of pre-specified properties. If the goal is to compare and evaluate response, then indicators must be easy to interpret, easy to compute and easy to translate to practical settings. If the goal is to monitor survey data collection, then indicators must be relevant and must allow for a top-down assessment of the impact of various population characteristics and survey design features. If the goal is to re-design survey data collection in order to improve representativity, then indicators must be effective.
With these goals and these requisites in mind, the project was divided in a theoretical part and an empirical part. The empirical part consisted of two designed data collection experiments.
As its major achievement RISQ has developed representativity indicators; simply termed as R-indicators and partial R-indicators. The indicators are documented, extensively tested and implemented as open source modules for R and SAS. The analyses, tests and experiments conducted within the project show that the indicators are valuable tools and fit for purpose. Survey designs tailored with the help of the indicators improved the representativity of response significantly. At workshops and conferences the project has advocated the use of the indicators as standards in quality reports. Besides in evaluating and designing survey data collection, the indicators also proved useful in monitoring the completion of registry data. The indicators are implemented in SAS and R open source modules. They can be downloaded from the website. Manual and test data sets are available.
Formally RISQ ended July 2010 with the release of software in SAS and R and the publication of theoretical and practical overview papers. Informal the project will be continued. In November 2010 release 2 of the software is planned that will add standard errors to partial R-indicators. In the spring of 2011 release 3 will include population-based R-indicators. The RISQ website, http://www.risq-project.eu will be updated on a regular basis and will contain the future releases. RISQ related questions and a request to receive future announcements can be send to risq@cbs.nl.
Contact persons for RISQ are:
- Statistics Netherlands: Barry Schouten (overall coordinator, bstn@cbs.nl) and Jelke Bethlehem (jbtm@cbs.nl)
- University of Southampton: Natalie Shlomo (N.Shlomo@soton.ac.uk) and Chris Skinner (C.J.Skinner@soton.ac.uk)
- Statistics Norway: Øyvin Kleven (Oyvin.Kleven@ssb.no)
- Katholieke Universiteit Leuven: Geert Loosveldt (Geert.Loosveldt@soc.kuleuven.be)
- Statistical Office of the Republic of Slovenia: Katja Rutar (Katja.Rutar@gov.si)
