COMBINE focused on the Netherlands, where social assistance benefits are indeed an important aspect of the welfare state, as approximately 400.000 individuals receive this benefit. The project was carried out in collaboration with Statistics Netherlands (CBS), and in particular with its department of Demographic and Socioeconomic Statistics, which is responsible for producing official statistics on social assistance benefits. The following activities were carried out:
1. The two registers containing independent data on social assistance benefits (municipal register and employment register) were linked at the individual level. Specifically, we linked data for individuals receiving these benefits sometime between January 2021 and December 2022. The dataset was expanded through December 2023 to include long-term benefit recipients. Basic descriptive analyses indicated measurement error in the data.
2. We estimated the appropriate statistical model to evaluate the size and the characteristics of measurement error in social assistance benefits. The statistical model we used is a hidden Markov model (HMM), a member of the family of latent-variable models. This model triangulates information from the two datasets and estimates the true state of receiving or not receiving social assistance benefits. Several HMMs with increasing complexity were estimated to select the optimal model for measurement error detection and correction. These models provided us with both aggregate and individual-level estimates of this error. Aggregate estimates include the size of this error in each dataset, the error-corrected estimates of the probability of receiving a social assistance benefit. Individual-level estimates include classification probabilities and diagnostics, i.e. on whether each individual is correctly or incorrectly observed as receiving or not receiving social assistance benefits in each of the two datasets.
3. The best-performing statistical model was further enriched with characteristics related to data collection (municipality size, software used for data collection) and individual characteristics of benefit recipients (gender, age, educational level, migration status and duration of receiving a benefit). This provided information on factors that are possibly related to increased measurement error or to groups where we observe more measurement error. This provided input for recommendations for improvement of data collection to the end users - data owners (municipalities and Employment Office).
The key findings of the project are:
1. The data used by Statistics Netherlands to produce statistics on social assistance benefits are of high quality. Overall, the Municipality indicator presents high agreement with the latent state over time, consistently around 98%.
2. The second register that provides information on social assistance benefits (Polisadministratie), we found in general high agreement. However, in January and December, several inconsistencies emerge. In December, we found lower agreement for the cases where the true state was not receiving benefits, and in January, we found lower agreement for the cases where the latent state was receiving benefits.
3. Considering the factors related to data collection, we found that the software used affects the agreement between the true state and the data used to produce official statistics, as certain software leads to lower agreement. However, the disagreement between the true and observed states never exceeds 5%. The level of agreement also varies modestly between municipalities. In a few municipalities, the level of agreement drops to 91%.
4. Considering population subgroups, we found higher inconsistencies between the true and observed state for higher vocational education graduates (HBO). For these groups, disagreement never exceeds 6%. No other individual characteristic produces notable differences in agreement between true and observed states.
5. Results of the analysis were not sensitive to the length of the observation period. When additional data were fed into the analysis, the main results remained practically unchanged.
All in all, the project illustrated that applying hidden Markov models to multisource data can produce improved statistics for social assistance benefits or serve as a powerful diagnostic tool for producing statistics using existing methods. This method can be easily applied in all cases of categorical statistics when multisource data is available.
