Final Report Summary - DISMEVAL (Developing and validating disease management evaluation methods for European health care systems)
Executive summary:
Aims and objectives
DISMEVAL aimed to contribute to developing new research methods and to generating the evidence base to inform decision making in the field of chronic disease management evaluation.
Specifically, its three-year programme of work sought to:
(1) review current approaches to chronic care management and their evaluations, as implemented by EU Member States at national and regional level,
(2) explore the policy context for chronic disease management in European countries,
(3) develop and validate evaluation methods for chronic disease management using data from existing programmes and approaches, and
(4) formulate recommendations for scientifically sound yet operationally feasible evaluation approaches for chronic disease management that are relevant to policies at EU and the wider European level and internationally.
Main findings
The DISMEVAL project has achieved these aims at three levels. First, we inform on the range of approaches to chronic care adopted in 13 European countries, and, by reflecting on facilitators and barriers to implementation, we provide policy makers and practitioners with a portfolio of options to advance care in a given policy context. We note how there remain considerable challenges towards the development of a policy framework for a strategic response to chronic disease, with considerable variation in the nature and scope of approaches to care across Europe. Challenges include a continued focus on complications management, with some movement towards more systematic disease management, and an overall lack of coordination between levels; failure to integrate risk minimisation and disease prevention with other components along the care spectrum; misalignment of financial incentives that tend to reward 'cure' over prevention; and a disjoint between intent, at national level, to enhance coordination and integration, and ability at regional or local level to translate these ambitions into practice. These observations emphasise the need for the development of a coherent response to chronic disease that takes account of the various tiers in the system and along the care continuum, with involvement of professionals forming a crucial component for achieving sustainable change.
Second, through testing and validating different evaluation methods using existing data we advance the research base in evaluation design and methodology, inform the design of future evaluations and enhance their value for decision-making. While acknowledging that a randomised controlled design should generally be considered as the most robust way of determining the effectiveness of a given intervention, the DISMEVAL project has identified and tested a wide range of methods for the evaluation of disease management that can be employed in situations where randomisation is not possible. We emphasise that rigorous evaluation is still possible even where baseline or predefined control groups are not available and how advanced designs can help better understand how different care components and processes might be effective for managing chronic disease in patients with different characteristics. Future evaluation work drawing on such approaches can provide insight into what works for who in disease management, a question that randomised trials have thus far been unable to answer. Project work further highlighted, through the introduction of statistical controls for selection or statistical matching, how findings were substantially different from simple comparisons of patients receiving a given disease management intervention and those who do not. Thus, the DISMEVAL project has shown how the use of randomisation or other methods of control is necessary to accurately assess the impact of such interventions. It also identified a range of methods that can be employed successfully to implement such controls.
Third, through the development of a report on recommendations for the evaluation of chronic disease management interventions DISMEVAL contributes to strengthening evidence-based policy and practice in Europe. The report is specifically targeted at policymakers, programme operators and researchers, explaining choices, options and trade-offs at methodological, practical and wider societal levels.
Challenges and lessons learned
Overall, evidence presented by DISMEVAL confirms the substantial heterogeneity across disease management interventions. This highlights how a given evaluation needs to be not only summative, but also include formative components to understand how interventions and programmes can be improved. Such consideration is particularly important against the background that the implementation of disease management is essentially a process of social change, it is important to combine quantitative data on effects with qualitative information concerning contexts. Use of mixed methods can ensure that disease management evaluation provides insight into how specific local conditions influence the outcomes of a given programme.
Work undertaken within DISMEVAL on evaluation metrics and methods was limited to disease-specific programmes, mirroring much of the existing research evidence that has focused on the management of a few specific diseases, such as diabetes. Interventions studied here tended to target those with good baseline status on key measures. There appeared to be little improvement on key scores for these baseline patients, and measures based on average improvements in scores tend to average those patients among whom improvement is possible with those among whom further improvement should not be expected. There is a need for further serious consideration to the identification of appropriate measures of success for disease management interventions that takes account of these differential impacts. Importantly however, there has been less focus on individuals with coexisting conditions or multiple health problems, even though it is this rapidly increasing population, with multiple disease processes and with diverse and sometimes contradictory needs, who pose the greatest challenge to health systems. Furthermore, as we have shown, the impact of chronic disease management interventions will depend, to a considerable extent, on the specific features of the healthcare setting within which they are introduced, and this observation seems to hold both within and between care systems. However, this work has shown how it can be possible to learn from the experiences of others.
Project Context and Objectives:
The rising burden of chronic disease presents challenges for all health systems.
[1] In the European Union, in 2006, between 20 and over 40 percent of the population aged 15 years and over reported a long-standing health problem and one in four currently receives long-term medical treatment.
[2] Other studies find the prevalence of common chronic disorders to be around 50 percent among adults aged 18 and older in seven high-income countries, including Germany, the Netherlands and the UK.
[3] Assessing the precise level, distribution and nature of the chronic disease burden in Europe remains a challenge;
[4]yet, it is clear that chronic diseases are an important cause of premature mortality and disability, greatly impacting on the years of life lived in good health. In high-income countries, depressive disorder, ischaemic heart disease and cerebrovascular disease, dementia, chronic obstructive pulmonary disease (COPD) and diabetes are among the ten leading contributors to the burden of disease
[5] with diabetes projected to rise further in importance during the next two decades, especially against the background of increasing levels of overweight and obesity.
[6]Chronic diseases pose a sizeable burden for national economies, with some studies estimating the associated costs at up to seven percent of a country's gross domestic product.
[7] Societal costs arise partly as a result of direct healthcare costs, including from healthcare utilisation, medication and potentially costly interventions, but these can also be caused by other factors, such as a decrease in work productivity.
[8] These challenges add to the complexity facing health systems, which require effective measures to prevent disease through reducing the major chronic disease risk factors and addressing influences that drive exposure
[9], while also providing services to meet the requirements caused by chronic health problems, ensuring that people with established illnesses can participate in society.
The DISMEVAL project
DISMEVAL (Developing and validating Disease Management Evaluation methods for European healthcare systems), a European collaborative project, aimed to contribute to developing new research methods, and to generate the evidence base to inform decision-making, in the field of chronic disease management evaluation.
The DISMEVAL project sought:
(i) to enhance our understanding of the use of various approaches to the evaluation of disease management in Europe and to identify examples of best practice and lessons learned;
(ii) to provide evidence-based recommendations to policymakers, programme operators and researchers on which evaluation approach is most useful in a given context;
(iii) to promote and support the networking and coordination of research and innovation activities relating to scientific knowledge and policy development in this area, building on existing work carried out in Member States and at the wider European level; and
(iv) to analyse scientific knowledge and developments as well as actions and policies within EU Member States, develop tools to assist policy analysis, and work in close collaboration with the Commission services, networks and experts in this area, and with stakeholder groups and various agencies to provide scientific input to support ongoing and planned actions and policies in the European Union.
The DISMEVAL project spanned multiple scientific domains by bringing together ten partners in seven European countries, representing a variety of disciplines including evaluation science, chronic care, disease management design and operations, epidemiology, economics, and health policy. Partner countries were selected to represent the variety of chronic disease management interventions that can be found across the EU, stretching from large-scale, population-based programmes to smaller, provider-centric interventions.
Within this broad context, DISMEVAL's three-year programme of work was developed around a set of key objectives:
(i) to review current approaches to chronic care implemented by EU Member States at national and regional level and to examine whether and how these interventions are being evaluated;
(ii) to enhance our understanding of the impact of macro-level health system features on chronic care interventions through exploring the policy context for chronic disease management in European countries;
(iii) to develop and validate methods for evaluating disease management programmes using data from existing programmes and approaches by employing a range of evaluation designs and assessing the sensitivity of findings to selected methods; and
(iv) to formulate recommendations for scientifically sound yet operationally feasible evaluation approaches for chronic disease management that are relevant to planned and ongoing policies at the EU and wider European level as well as internationally.
This was to be achieved through a programme of work that can be differentiated into three phases:
(1) review of current approaches to the implementation and evaluation of chronic disease management;
(2) testing and validation of methods and metrics for the evaluation of chronic disease management; and
(3) development and recommendations for methods and metrics for the evaluation of chronic disease management. Here we briefly outline each of the three phases and the work packages carried out therein.
Phase 1: Review of current approaches to the implementation and evaluation of chronic disease management in Europe
Phase 1 comprised work packages two to four. Work package 2 (WP2) 'Approaches to chronic disease management' sought to review approaches to managing chronic conditions that have been developed and/or implemented in different countries in Europe, and to assess whether and how countries evaluate approaches to chronic disease management, so feeding into work package 3 (WP3) 'Assessment of disease management evaluation approaches in Europe'. Thus, based on data collected in WP2, WP3 aimed to provide an overview of the types of evaluation approaches that are being used in Europe to assess the impact of structured approaches to disease management on the cost and quality of chronic illness care. Both work packages informed WP4 'Assessing the policy context for chronic disease management in Europe', which aimed to assess the overall policy framework for chronic disease management in selected European countries.
Phase 2: Testing and validation of methods and metrics for the evaluation of chronic disease management
Phase 2 of the project included work packages 5-10. The main aim of this phase was to utilise data from existing chronic disease management programmes, or their equivalent, in partner countries, to test and validate different evaluation options reviewed in Phase 1 of the project. The countries included were Austria (WP5), Denmark (WP6), Germany (WP7), France (WP8), the Netherlands (WP9), and Spain (WP10).
Phase 3: Development and recommendations for methods and metrics for the evaluation of chronic disease management
Phase 3 comprised work package 11, which sought to summarise the findings and to present best practice and lessons learned from work undertaken in WPs 5-10. It further aimed to present validated recommendations on performance indicators and evaluation methods for disease management programmes or their equivalent. The final output of this work package is a report on evaluation of chronic disease management that outlines choices, options and trade-offs to policymakers, programme operators and researchers, and with recommendations presented alongside their rationale.
References
[1] Yach D, Hawkes C, Gould C, Hofman K. The global burden of chronic diseases. JAMA 2004;291:2616-22.
[2] TNS Opinion and Social. Health in the European Union. Special Eurobarometer 272e. Brussels: European Commission, 2007.
[3] Schoen C, Osborn R, Doty M, Bishop M, Peugh J, Murukutla N. Towards higher-performance health systems: adults' health care experiences in seven countries. Health Affairs 2007;26:w717-w34.
[4] Pomerleau J, Knai C, Nolte E. The burden of chronic disease in Europe. In: Nolte E, McKee M, editors. Caring for people with chronic conditions. A health system perspective. Maidenhead: Open University Press, 2009:15-42.
[5] World Health Organization. The global burden of disease: 2004 update. Geneva: World Health Organization, 2008.
[6] Finucane M, Stevens G, Cowan MD, G, Lin J, Paciorek C, Singh G, et al. National, regional, and global trends in body-mass index since 1980: systematic analysis of health examination surveys and epidemiological studies with 960 country-years and 9.1 million participants. Lancet 2011;377:557-67.
[7] Suhrcke M, Nugent R, Stuckler D, Rocco L. Chronic disease: an economic perspective. Oxford: Oxford Health Alliance, 2006.
[8] Suhrcke M, Fahey D, McKee M. Economic aspects of chronic disease and chronic disease management. In: Nolte E, McKee M, editors. Caring for people with chronic conditions: A health system perspective. Maidenhead: Open University Press, 2008:43-63.
[9] Novotny T. Preventing chronic disease: everybody's business. In: Nolte E, McKee M, editors. Caring for people with chronic conditions: A health system perspective. Maidenhead: Open University Press, 2008:92-115.
[10] Mattke S, Seid M, Ma S. Evidence for the effect of disease management: Is $1 billion a year a good investment? Am J Manag Care 2007;13:670-676.
Project Results:
This section presents the main findings of work carried out within the DISMEVAL project. It first reviews current approaches to the implementation and evaluation of chronic disease management. Second, it reports on the testing and validation of methods and metrics for the evaluation of chronic disease management. The third stage involved the development and recommendations for methods and metrics for the evaluation of chronic disease management; we here reflect on some of the challenges identified, which may be deemed necessary to consider for users, funders and researchers interested in the evaluation of structured approaches to chronic disease management. It concludes with identifying continued challenges and lessons learned. A more detailed account of the findings of the DISMEVAL project is documented in the Final Report, with an accompanying volume providing recommendations for funders and users on methods and metrics for the evaluation of chronic disease management. Both reports are available at http://www.dismeval.eu/publications/.
Reviewing approaches to chronic disease management in Europe
The review of approaches to chronic disease management in Europe drew on a common template for data collection, which was based on a structured questionnaire used in the framework of a previous study by Nolte, Knai and McKee (2008)[1], and informed, to a great extent, by the Chronic Care Model (CCM) developed by Wagner and colleagues in the United States.[2] Accordingly, the template sought to gather information on:
(i) the health system and policy context;
(ii) the type and format of approaches to managing chronic disease;
(iii) the evaluation of existing approaches; and
(iv) system markers of success or failure for organisational approaches to chronic disease management.
Selection of countries for review was guided by three main criteria in order to capture the range of approaches to funding and governing health care across Europe; the range of stages in economic development; and geographical spread across the EU. On this basis, we selected 13 countries for detailed analyses: Austria, Denmark, England, Estonia, France, Germany, Hungary, Italy, Latvia, Lithuania, the Netherlands, Spain and Switzerland (the only non-EU country). Principal data collection was carried out by key informants in each country over the period of June 2009 to December 2009, with two rounds of updates in autumn 2010 and summer 2011.
Based on this survey of approaches to chronic disease management in 13 European systems we found that most countries have sought to create a regulatory and policy framework to respond to chronic disease during recent years. These generally aim to promote approaches that better integrate care and improve coordination between sectors and levels of care but countries differ with regard to their vision towards controlling and managing chronic disease. Likewise, approaches to chronic disease management vary in scope and nature across Europe.
While our review did not attempt to present a comprehensive inventory of all approaches being implemented in a given country, some of the key observations were that:
- the majority of countries tend to focus on care models for populations with defined conditions, most frequently diabetes type 2, and involve some form of GP-led care coordination
- nurse-led approaches are becoming more common although there are differences in the degree to which nurses can operate independently
- patient access is typically granted in line with access to usual care, although many approaches are being implemented in selected geographical regions so potentially limiting access to defined population groups
- the majority provide some form of patient self-management support, although the level and scope of support offered varies
- the overall the use of clinical information systems for chronic disease management tends to be the least developed strategy in most approaches
Based on the survey data collected within DISMEVAL, we also examined whether and how chronic disease management models and programmes, or their equivalent that have been or are being implemented in European countries, are being evaluated. We find that while most initiatives reviewed had undergone some form of evaluation or had evaluation plans in place, the nature and scope of evaluations varied, with differences in objectives, design, the performance metrics used, the length of observation and approaches to data collection. We identified a range of challenges posed to the more systematic use of evaluation of complex healthcare interventions such as disease management in European health systems, including a perceived lack of an evaluation culture in some settings, alongside lack of financial and human resources to conduct systematic evaluation. We note that evaluation of approaches to disease management reviewed would likely benefit analytically from increased use of sophisticated statistical techniques, but also conceptually from drawing more explicitly on one of the many possible theories of behaviour change to better link the choice of performance measures to the goals of the intervention (and of the evaluation). There may also be scope to more systematically draw in mixed-methods approaches to help place observed quantitative findings into the context within which the intervention under evaluation is embedded. More information is needed about the characteristics of the intervention and its intended populations, requiring greater specification of the wider context so as to improve comparisons and potential transferability across settings and countries in Europe.
In order to place the survey findings reviewed in the preceding sections into the wider context, we carried out interviews with key informants in a sample of countries that formed the core group of the DISMEVAL project (Austria, Denmark, France, Germany, the Netherlands and Spain). This provided an opportunity to learn how representatives from different sectors are approaching chronic care, and their perception of the policy framework for providing a strategic response to chronic disease. Some of the reported challenges included a continued focus of chronic care on complications management, with some movement towards more systematic disease management, and an overall lack of coordination between levels; failure to integrate risk minimisation and disease prevention with other components along the care spectrum; misalignment of financial incentives that tend to reward 'cure' over prevention; and a disjoint between intent, at national level, to enhance coordination and integration, and ability at regional or local level to translate these ambitions into practice. While some of these observations are perhaps not specific to chronic care as such, they emphasise the need for the development of a coherent response to chronic disease that takes account of the various tiers in the system and along the care continuum, with involvement of professionals, who exert a large degree of control in healthcare organisations such as primary care practices and hospitals. As we and others have argued elsewhere, failure to engage them in the reform process is likely to hamper sustainable change.[3-5]
Testing and validation of methods and metrics for the evaluation of chronic disease management
The testing and validation of methods and metrics for the evaluation of chronic disease management comprised studies carried out in six countries and using data from existing interventions. These were: disease management programmes for diabetes type 2 in Austria and Germany, diabetes care groups in the Netherlands, provider networks for diabetes and for cancer in France, an interdisciplinary and sectoral rehabilitation programme for people with chronic obstructive pulmonary disease and for diabetes in Denmark, and a nurse-led intervention targeting a working-age population at risk of cardiovascular disease in Spain. All interventions were implemented in a non-experimental setting; the only exception was the diabetes disease management programme in Salzburg, Austria, which was implemented as a pragmatic cluster-randomised controlled study.
As interventions and the setting in which they were implemented varied, so did their approaches to testing and validation approaches to evaluation. Thus, country studies aimed to:
- quantify differences in effect sizes of structured care within a diabetes disease management programme using randomised and non-randomised controlled and non-experimental designs (Austria)
- test different approaches to identify treatment-control matches in non-experimental settings and quantify the likely impact on the effect estimate of an interdisciplinary and sectoral intervention for patients with chronic obstructive pulmonary disease or diabetes (Denmark)
- compare different methods to adjust for confounding in a non-experimental setting using routine data to assess intervention effect of a diabetes disease management programme (Germany)
- test for selection bias for participating in a structured care programme for diabetes (Germany, France )
- employ advanced methods of disease management evaluation in non-experimental settings to better understand differential effects of structured care components on subpopulations (Netherlands, Spain).
Secondary goals of each analysis further included evaluating the effects of the intervention under study on patient outcomes more generally (all countries), alongside better understanding of the usefulness of current approaches to evaluation in the context of intervention practice (France, Netherlands) and to derive recommendations for further development of interventions and evaluation practice. We here provide an overview of the main findings of country reports.
Evaluating the diabetes disease management programme in Austria using an uncontrolled design overestimates treatment effects
In Austria, the evaluation of the effect of the diabetes disease management programme 'Therapie Aktiv', using a randomised controlled design, found a reduction in HbA1c levels of 0.13 percent after one year. This effect was not statistically significant while measures of process quality such as regular eye and foot examination and patient education improved significantly. In contrast, using an uncontrolled before-after design, treatment effect was estimated at a significant reduction in HbA1c levels of 0.41 percent in the intervention group. Extrapolating these findings to clinically relevant endpoints such as number needed to treat to avoid one case of myocardial infarction or one diabetes complication over a period of ten years, the uncontrolled before-after design overestimated the treatment effect by a factor of three. These findings support the general notion that use of a randomised controlled design should be considered as the main means for evaluating treatment effect of a structured care intervention. Cluster randomisation as applied in the case of Austria can be seen to provide a pragmatic approach to DMP evaluation where a randomised controlled design is not feasible.
Different approaches to identify treatment-control matches in a non-experimental intersectoral intervention for patients with chronic obstructive pulmonary disease in Denmark provide different estimates of treatment effect
In Denmark, the evaluation of the effect of an interdisciplinary and intersectoral rehabilitation programme for patients with chronic obstructive pulmonary disease (COPD) found effect sizes to vary with the method of constructing control groups for the intervention-control and the difference-in-difference analysis. For example, propensity score matched sampling lowered the magnitude of the predicted intervention effect for COPD-specific hospital bed days when compared with controls created by random sampling. Likewise, control groups not matched by disease severity overestimated the effects of pulmonary rehabilitation on COPD-specific hospital contacts and bed days.
The study considered the method of control group construction and matching using propensity scoring and the use of difference-in-difference analyses to assess intervention effect to be optimal to evaluate the impact of interventions in a non-experimental setting. As for the impact of the overall programme, the case study provided evidence that the intervention might have decreased the pace of disease progression in the intervention group, which was reflected in a non-significant increase in COPD hospital contacts, bed days, ambulatory visits and emergency room visits in the intervention group while these indicators significantly increased in the entire sample.
Different methods to adjust for baseline differences in a non-experimental setting using routine data to assess intervention effect of a diabetes disease management programme in Germany result in similar effect measures
In Germany, the evaluation of intervention effect of a diabetes disease management programme (DMP) in a non-experimental setting found that different matching and/or weighting methods used resulted in similar effect measures for the outcome variables analysed. It highlighted how the applicability of available statistical methods and tools to perform a sound evaluation of programme effects is conditional on the availability of reliable baseline data prior to enrolment in the intervention, to enable adjustment of differences between intervention and control group. Therefore, a great effort should be made to collect detailed and valid data to maximise the usefulness of routine data for evaluation.
As for the impact of the overall programme, the case study confirmed the findings of other studies that participation in the diabetes DMP improved process parameters in diabetes care especially those related to the monitoring of the disease. However, intensified care in the programme was accompanied by higher overall costs, primarily because of higher prescription costs. In order to draw valid conclusions about DMP effects on clinical endpoints such as mortality or micro- and macrovascular complications, a longer study period consistent with what is known about the time course of the disease should be chosen.
Patient selection for participating in a structured care programme for diabetes in Germany and France leads to over- and/or underestimation of findings of effect of the intervention
The evaluation of the diabetes disease management programme in Germany also observed a significant reduction of mortality in the intervention group. This effect was likely attributable to GPs systematically excluding patients from joining the programme who were more likely to die in the near future. Adjusting for baseline variables reduced this effect compared with the unadjusted analysis. Future analyses should include further adjustment for variables suited to predict short-term mortality risk. Furthermore, a longer observation period would be required to assess whether the mortality difference diminishes over the years.
Evidence of patient selection into the intervention was also observed in the case study for France, which examined patient characteristics of those enrolled with a diabetes provider network. These patients were found to be of younger age whose diabetes was diagnosed more recently but showed evidence of worse glycaemic control than patients in the reference population. By comparing a standard uncontrolled pre-post evaluation design with a pre-post design after calibration with the reference population at baseline, the analysis further showed that the uncontrolled evaluation design overestimated intervention effect on a number of clinical outcomes, including improvements in HbA1c levels and body mass index while underestimating deterioration in renal function in diabetic patients.
Advanced methods of disease management evaluation in non-experimental settings in the Netherlands and Spain help understand the differential effects of care components on subpopulations and so inform further development of structured care approaches
Given that most randomised controlled trials are conducted in academic settings and provide limited insight into the impact of disease management in the everyday practice of health care, in the Netherlands, the evaluation of integrally financed, regional disease management programmes for diabetes used a variety of approaches to better understand the effects of different programmes. It found that applying methods that permit for subgroup analyses provide important new insights into differential component effects. Thus, while the overall analysis of intervention effects found only modest impacts of the care programmes on the health status of patients with diabetes type 2, subgroup analyses revealed disease management to be considerably more effective for patients with poor baseline clinical values. As the vast majority of patients included in the analyses had good baseline values of most clinical endpoints, this differentiation provided a plausible explanation for the modest overall effects of the intervention. This suggests that further development of the intervention should move towards a more tailored approach to diabetes care, in which the characteristics of patients directly determine the processes of diabetes care, including self-management support. Such a move will, however, require improvements in the current systems for data registration to provide valid and reliable information on patients' health status to determine care intensity.
Similarly, in assessing the effect of a nurse-led intervention targeting a working-age population at risk of cardiovascular disease in Spain, the use of advanced methodological approaches that permit for subgroup analysis in a non-experimental setting provided important new insights into the effects of the intervention.
Taken together, observations emerging from the six country reports suggest that in non-experimental settings, the creation of statistical controls or adjustment using propensity or matching methods was feasible using routine data sets. Application of such methods consistently provided different estimates, generally smaller, of intervention effectiveness. While the literature suggests that more aggressive methods such as boosted regression may provide a more suitable means for matching, results from studies presented here that compared alternative matching methods found results to be similar. Analyses presented here also provided insights into programme effectiveness without introducing an external control by examining variation across interventions such as in the Netherlands, or by excluding patient populations close to the inclusion threshold (Spain).
Practical considerations for evaluation
Data availability constitutes a considerable challenge to evaluation research, even where evaluation design and capacity is of high quality. Where routinely collected data is used, this may be incomplete and may require systematic scrutiny to assess the implications of missing data for the analysis and interpretation of findings; this may require additional resources where missing data has to be imputed from elsewhere. Even where routine data is fairly complete, it may be inadequate for the purpose of evaluation as it is typically used for administrative purposes only and may not record outcomes of interest. Evaluations may therefore necessitate new data collection, with consequent resource implications.
Evidence of work carried out within the DISMEVAL project has highlighted the many opportunities and challenges arising from the use of routinely collected data for the evaluation of disease management. Routine data have the advantage of relatively easy access and data are available for a large patient sample, capturing information on daily practice of healthcare provision and permitting assessment of relatively long-term outcomes, such as mortality within a limited timeframe. Furthermore, financial costs of data collection tend to be low. These advantages potentially outweigh the challenges associated with the considerable volume of missing data that tends to be common, in particular for data collection systems that are newly introduced and/or for which validity checks are not routinely carried out, as has been observed for the German case study which drew on routinely collected data from a statutory health insurance fund. Therefore, any evaluation that makes use of routine data systems will have to allow sufficient time to enable validity checks and a good understanding of the context within which data collection takes place, in particular in relation to the range of outcome variables to be considered for evaluation where data is not collected for research but for administrative purposes. Experiences from the Dutch and the Danish case studies stressed that where new data documentation on disease management interventions is being established, this will require supportive infrastructure for data to be usefully utilised for evaluation. The example from France has illustrated how heterogeneity of routine data collection systems can pose challenges for their use in evaluation, highlighting the need for greater collaboration between those responsible for data collection.
The experiences of DISMEVAL case studies further suggest that even where routine data can be accessed in principle, full exploitation of data available may be compromised, which had been the case for data from statutory insurance funds in Germany and Austria. While maintaining data confidentiality is a serious concern in any research, there may be a need for decision-makers to consider putting safeguards in place to allow for full access of routinely collected data sets while maintaining high standards of data privacy and confidentiality, such as is illustrated in the Danish case study. An important lesson drawn from all case studies presented in DISMEVAL is the necessity, in the evaluation plan, to allow for sufficient time to negotiate access to data and, where necessary, to set sufficient additional resources aside to allow for data extraction by the holder of the data.
Challenges and lessons learned
The DISMEVAL project set out to enhance our understanding of the use of various approaches to the evaluation of disease management in Europe, to identify examples of best practice and lessons learned and to provide evidence-based recommendations to policymakers, programme operators and researchers on evaluation approaches that may be most useful in a given context.
The work carried out within DISMEVAL set out on the premise that experimental research designs, particularly randomised controlled trials, are generally considered to be the most rigorous way of determining the effectiveness of a given intervention.[6] The Austrian case study in DISMEVAL described above has illustrated how it can be feasible to employ a randomised design in routine settings where the context allows for such a design to be applied. However, use of an experimental design may not be possible (or desirable) for population-wide disease management interventions, which are frequently implemented in an operational setting and do not usually have a control group available, such as in Germany and the Netherlands. For example, in the Dutch case study use of an experimental approach was not possible due to the nation-wide roll-out of structured care approaches for diabetes and the unsuitability of using historic controls. Furthermore, although randomised studies are generally considered to form the most rigorous means to assess intervention effect, the scientific rigour of required designs limits the generalisability of findings to larger and inherently more heterogeneous populations of, for example, chronically ill patients. Selection bias poses a threat to randomised designs as it does for non-randomised designs as we have highlighted.
Observational study designs are more suitable for 'real-world' disease management evaluations, keeping their methodological limitations in mind. Given that disease management is essentially a population-based care strategy, advancing observational study designs is crucial to come to strong conclusions on how to best target subgroups of chronically ill patients in the daily healthcare practice. The DISMEVAL project has identified and tested a wide range of methods that can be employed in situations where randomisation is not possible, emphasising that rigorous evaluation is still possible even where baseline or predefined control groups are not available. Project work further highlighted, through the introduction of statistical controls for selection or statistical matching, how findings were substantially different from simple comparisons of patients receiving a given disease management intervention and those who do not. Thus, the DISMEVAL project has shown how the use of randomisation or other methods of control is necessary to accurately assess the impact of such interventions. It also identified a range of methods that can be employed successfully to implement such controls.
Different (combinations of) care components and processes might be effective for managing chronic disease in patients with varying age, disease duration, health status, co-morbidity, education level, socio-economic status and so on. Contrary to most methods aimed at evaluating disease management interventions, which focus on assessing a single treatment effect, meta-analysis and meta-regression allow for investigations of which patient groups will benefit most from which treatment. Therefore future evaluation work drawing on such approaches can provide insight into what works for whom in the area of disease management, a question that randomised trials thus far have been unable to answer.[7] In addition, meta-regression analyses can be adjusted for baseline (prognostic) factors, which can increase the power of the analysis to detect a true treatment effect and allows adjustment for confounding factors, which is a particular advantage for analyses of observational data.
Overall, evidence presented by DISMEVAL confirms the substantial heterogeneity across disease management interventions. This highlights how a given evaluation needs to be not only summative, but also include formative components to understand how interventions and programmes can be improved. Such consideration is particularly important against the background that the implementation of disease management is essentially a process of social change, it is important to combine quantitative data on effects with qualitative information concerning contexts. Use of mixed methods can ensure that disease management evaluation provides insight into how specific local conditions influence the outcomes of a given programme.
Work undertaken within DISMEVAL on evaluation metrics and methods was limited to disease-specific programmes, mirroring much of the existing research evidence that has focused on the management of a few specific diseases, such as diabetes. Interventions studied here tended to target those with good baseline status on key measures. There appeared to be little improvement on key scores for these baseline patients, and measures based on average improvements in scores tend to average those patients among whom improvement is possible with those among whom further improvement should not be expected. There is a need for further serious consideration to the identification of appropriate measures of success for disease management interventions that takes account of these differential impacts. Importantly however, there has been less focus on individuals with coexisting conditions or multiple health problems[8-9], even though it is this rapidly increasing population, with multiple disease processes and with diverse and sometimes contradictory needs, who pose the greatest challenge to health systems.[10] Furthermore, as we have shown, the impact of chronic disease management interventions will depend, to a considerable extent, on the specific features of the healthcare setting within which they are introduced, and this observation seems to hold both within and between care systems. However, this work has shown how it can be possible to learn from the experiences of others.
References
[1] Nolte E, Knai C, McKee M, editors. Managing chronic conditions: Experience in eight countries. Copenhagen: World Health Organization on behalf of the European Observatory on Health Systems and Policies, 2008.
[2] Wagner E. Chronic disease management: What will it take to improve care for chronic illness? Effective Clinical Practice 1998;1:2-4.
[3] Nolte E, McKee M. Making it happen. In: Nolte E, McKee M, editors. Caring for people with chronic conditions: A health system perspective. Maidenhead: Open University Press, 2008:222-244.
[4] Ham C. Improving the performance of health services: the role of clinical leadership. Lancet 2003;361:1978-1980.
[5] Nolte E, Knai C, Hofmarcher M, Conklin A, Erler A, Elissen A, Flamm M, Fullerton B, Sonnichsen A, Vrijhoef HJM. Overcoming fragmentation in health care: chronic care in Austria, Germany and the Netherlands. Health Econ Polit Law 2012;7:125-146.
[6] Conklin A, Nolte E. Disease management evaluation. A comprehensive review of current state of the art. Santa Monica: RAND Corporation, 2011.
[7] Shojania K, Ranji S, McDonald K, Grimshaw J, Sundaram V, Rushakoff R, et al. Effects of quality improvement strategies for type 2 diabetes: a meta-regression analysis. JAMA 2006;296:427-40.
[8] Piette J, Kerr E. The impact of comorbid chronic conditions on diabetes care. Diabetes Care 2006;29:725-31.
[9] Struijs J, Baan C, Schellevis F, Westert G, van den Bos G. Comorbidity in patients with diabetes mellitus: impact on medical health care utilization. BMC Health Serv Res 2006;6:84.
[10] Nolte E, McKee M. Caring for people with chronic conditions. A health system perspective. Copenhagen: World Health Organization on behalf of the European Observatory on Health Systems and Policies, 2008.
Potential Impact:
The DISMEVAL project contributes to advancing the evidence base and informing decision making in addressing chronic illness in Europe at three levels. First, we inform on the range of approaches to chronic care adopted in 13 European countries, reflecting on facilitators and barriers to implementation. It provides policy makers and practitioners with a portfolio of options to advance chronic care approaches in a given policy context. We note how that there remain considerable challenges towards the development of a policy framework for providing a strategic response to chronic disease, with considerable variation in the nature and scope of approaches to chronic disease management across Europe. Challenges included a continued focus of chronic care on complications management, with some movement towards more systematic disease management, and an overall lack of coordination between levels; failure to integrate risk minimisation and disease prevention with other components along the care spectrum; misalignment of financial incentives that tend to reward 'cure' over prevention; and a disjoint between intent, at national level, to enhance coordination and integration, and ability at regional or local level to translate these ambitions into practice. These observations emphasise the need for the development of a coherent response to chronic disease that takes account of the various tiers in the system and along the care continuum, with involvement of professionals forming a crucial component for achieving sustainable change.
Second, through testing and validating different evaluation methods using existing data we advance the research base in evaluation design and methodology, inform the design of future evaluations and enhance their value for decision-making. Thus, while acknowledging that a randomised controlled design should generally be considered as the most robust way of determining the effectiveness of a given intervention, the DISMEVAL project has identified and tested a wide range of methods for the evaluation of disease management interventions that can be employed in situations where randomisation is not possible. We emphasise that rigorous evaluation is still possible even when baseline or predefined control groups are not available, and how advanced designs can help better understand how different care components and processes might be effective for managing chronic disease in patients with different characteristics. Future evaluation work drawing on such approaches can provide insight into what works for who in the area of disease management, a question that randomised trials have thus far been unable to answer. Project work further highlighted, through the introduction of statistical controls for selection or statistical matching, how findings were substantially different from simple comparisons of patients receiving a given disease management intervention and those who do not. Thus, the DISMEVAL project has shown how the use of randomisation or other methods of control is necessary to accurately assess the impact of such interventions. It also identified a range of methods that can be employed successfully to implement such controls.
Third, through the development of a report on recommendations for the evaluation of chronic disease management interventions, it contributes to strengthening evidence-based policy and practice in Europe. The report is based on the premise that policymakers, programme operators and researchers need validated approaches for the evaluation of disease management to design effective, efficient and equitable interventions to improve the care for people with chronic conditions.
Of particular relevance there are two components of evaluation:
(1) the performance indicators that are used to capture the impact of a given programme and
(2) the attribution strategy, which enables identifying any observed changes in selected indicators as programme effect rather than as a consequence of other factors, such as secular trends or changes in the treatment (ie establishing the counterfactual - whether the change could have been observed in the absence of the programme).
These methods need to be scientifically sound but also operationally feasible to address common threats to validity in disease management evaluation.
The report is therefore targeted at policymakers, programme operators and researchers, explaining choices, options and trade-offs for the evaluation of disease management based on analyses undertaken within the DISEMEVAL project. At the outset it is important to note that many of the issues discussed in this report are not specific to disease management evaluation but can be seen to apply to any evaluation of complex interventions in healthcare.
However, there are specific concerns around evaluation design and metrics that are of relevance to disease management evaluation and which are given particular attention, including:
(i) the context for evaluating disease management, exploring the reasons for undertaking evaluation in the first place and explaining some of the underlying principles for doing so;
(ii) the methods and metrics of disease management evaluation, focusing specifically on themes that have emerged as being pertinent to work carried out within the DISMEVAL project;
(iii) practical considerations for disease management evaluation, based on experience of work undertaken in DISMEVAL; and
(v) the broader challenges and lessons learned that may be relevant for policymakers, funders and practitioners interested in the use and usefulness of disease management evaluation more generally.
We therefore expect this publication to provide a major resource to guide the evaluation of disease management interventions in European settings and so contribute to strengthening the evidence-base required to inform the selection of efficient and effective interventions to address the growing burden of chronic disease in Europe.
Beyond this report which is, as noted above, specifically targeted at policymakers, programme operators and practitioners, work carried out within the DISMEVAL project has been disseminated widely, both at national and international levels. Progress on work and key findings were primarily communicated through presentations in the context of meetings and conferences targeting researchers, policy makers and practitioners in Europe, the Americas, India and Australia; with peer-reviewed publications becoming a more important source for communication especially in the third and final year of the project. We expect dissemination of project findings to continue beyond the formal completion of the DISMEVAL project, both through presentations and discussions at (inter)national conferences and meetings and, in particular, publications, both in the academic literature as well as in the form of policy briefs and monographs targeting the policy and practitioner community in particular.
Key dissemination achievements during the lifetime of the DISMEVAL project included:
1. The organisation and presentation of DISMEVAL workshops at European conferences:
- Presentation of the DISMEVAL project at the Working conference Health Services Research in Europe, the Hague, the Netherlands, 8-9 April 2010. This conference provided a platform to present the project to a wider audience of health services researchers and policy makers in Europe
- A workshop 'Improving chronic illness care: experiences with different approaches internationally' at the 27th ISQua International Conference, Paris, France, 10 October 2010. This workshop sought to discuss current evidence of how disease management and other innovations impact chronic conditions and how health system characteristics influence the impact of such interventions. It involved presentations by three DISMEVAL partners (RAND, GUF, UM)
- A DISMEVAL workshop organised by the DISMEVAL Consortium at the 3rd European Public Health Conference 'Integrated Public Health', Amsterdam, the Netherlands, 11 November 2010. The workshop aimed to provide a forum for discussion of conceptual, methodological and policy-related challenges of the evaluation of approaches targeting people with chronic conditions, and so informing the development of similar undertakings elsewhere. It was coordinated and chaired by the coordinator (RAND) with presentations by DISMEVAL partners from Austria (PMU), Denmark (UCPH), Germany (GUF; AQUA), France (UPVM), the Netherlands (UM) and Spain (ISCIII)
- A DISMEVAL workshop organised by the DISMEVAL Consortium at the 17th WONCA Europe 2011 Conference, Warsaw, Poland, 9 September 2011. The workshop aimed to discuss findings of a range of DISMEVAL work packages. It was coordinated and chaired by the Austrian DISMEVAL partner (PMU) with presentations by DISMEVAL partners from Austria (PMU), Germany (GUF), France (UPVM) and Spain (ISCIII)
2. Presentation of the project and/or individual work packages at national conferences including
- Annual Research Meeting of the German Society for Social Medicine (DGSMP) and the German Society for Medical Sociology (DGMS), Hamburg, Germany, 23-25 September 2009 (RAND)
- 2nd Conference on Chronic Care, Santiago de Compostela, Spain, 25-26 February 2010 (RAND)
- 2nd Forum Internacional Sobre Doentes Cronicos, Lisbon, Portugal, 1 May 2010 (RAND)
- Annual Danish Public Health Society Meeting 'Sundhedsdage 2010', Nyborg, Denmark, 29-30 June 2010 (UCPH)
- 44th Conference of the German Society of General Medicine DEGAM, Dresden, Germany, 23-25 September 2010 (PMU, GUF)
- 16th annual meeting of the Danish Forum for Health Services research, Copenhagen, Denmark, 25 November 2011 (UCPH)
- Austrian, German and Southern Tyrolian Society of General Medicine DEGAM, Salzburg, 21 September 2011 (PMU, GUF)
- GMDS/DGEpi 2011: Annual Meeting of the German Society for Medical Informatics, Biometrics and Epidemiology (GMDS) and the German Society for Epidemiology (DGEpi), Mainz, Germany, 26 September 2011 (AQUA)
- German Conference of Health Services Research, DKVF, Cologne, Germany, 20 October 2011 (AQUA)
- Chronic Disease and Health Management - 3rd edition 'Investing in the quality of health care', Bucharest, Romania, 7 October 2011 (RAND)
- Congress of the French society for Public Health, Lille, 2-4 November 2011 (UPVM)
3. Presentation of the project and/or individual work packages at international conferences including
- Presentation of the DISMEVAL project at the Academy Health Annual Research Meeting, Chicago, USA, 26-20 June 2009. This conference presented the first opportunity to present the aims and objectives of the DISMEVAL project to a wider audience of health services researchers and policy makers in North America, also acting as a catalyst to communicate European health services research to the research and policy community outside Europe (RAND)
- 2010 conference of 'The Future of Primary Health Care in Europe III', European Forum for Primary Care, Pisa , Italy, 8 August 2010 (poster: GUF)
- EURO-EPI 2010 'Epidemiology and Public Health in an Evolving Europe', Florence. Italy, 8 November 2011 (GUF)
- 4th European Public Health Conference 'Public Health and Welfare', Copenhagen, Denmark, 10-12 November 2011 (UCPH)
4. Invited (keynote) lectures at national and international meetings and conferences
- Hemispheric Meeting of the Social Protection and Health Network; Improving Chronic Disease Prevention and Management in Latin America and the Caribbean, Santiago de Chile, Chile, 30 September 2010. This meeting provided the unique opportunity to communicate and discuss findings emerging from DISMEVAL to senior policy makers in Latin America and the Caribbean, further acting as a means to communicate European research to the policy community outside Europe. (RAND)
- Belgian Presidency of the European Council 'Innovative approaches for chronic illness in public health and healthcare systems', Brussels, Belgium, 20 October 2010. This meeting, organised under the Belgian Presidency provide the opportunity to communicate and discuss findings emerging from the DISMEVAL to senior policy makers in the European Union (RAND)
- 3rd European Public Health Conference 'Integrated Public Health', Amsterdam, the Netherlands, 13 November 2010. This dissemination opportunity involved a plenary presentation at Europe's main conference for public health researchers, which brings together the public health research and policy community (RAND)
- International Seminar 'Becoming the Best: Building Sustainability High Performing Health Systems', Institute of Health Economics, Alberta, Canada, 15 April 2011 (RAND (P1))
- European Health Management Annual Conference 'Integration in Health and Health Care', Porto, Portugal, 23 June 2011. The annual EHMA conference brings together research, policy and management communities in the field of health management, so providing an opportunity. The dissemination opportunity involved a keynote lecture, so providing a means to communicate DISMEVAL findings to the health management community in particular (RAND)
- National Summit on Non communicable Diseases, Ministry of Health and Family Welfare and WHO Country Office for India, New Delhi, India, 23-24 August 2011. This national summit provided the unique opportunity to communicate and discuss findings from DISMEVAL to senior policy makers and researchers in India, further acting as a means to communicate European research to the research and policy community outside Europe. (RAND)
- Annual Conference Australian Disease Management Association, Canberra, Australia, 26 August 2011. This national conference provided the unique opportunity to communicate and discuss findings from DISMEVAL to senior policy makers and researchers in Australia, further acting as a means to communicate European research to the research and policy community outside Europe. (UM)
In addition to communicating emerging findings from the DISMEVAL project through presentations, peer-reviewed publications form an important route for dissemination. During the lifetime of the project, DISMEVAL partners have begun publishing in peer-reviewed journals.
These included:
1. Fullerton B, Nolte E, Erler A. Qualität der Versorgung chronisch Kranker in Deutschland. Z Evid Fortbild Qual Gesundhwes 2011;105:554-562. [Article in German]
2. Elissen AMJ., Duimel-Peeters IGM, Spreeuwenberg C, Vrijhoef HJM. Dismeval: Europees onderzoek naar 'best practices' op het gebied van disease management evaluatie. Tijdschrift voor Gezondheidswetenschappen 2011; 89:173-177. [Article in Dutch]
3. Flamm M, Panisch S, Winkler H, Sönnichsen AC. Impact of a randomized control group on perceived effectiveness of a Disease Management Programme for diabetes type 2. Eur J Public Health 2011 Oct 11 [Epub ahead of print]
4. Jacobsen R, Frølich A, Godtfredsen NS. Impact of exercise capacity on dyspnea and health-related quality of life in COPD patients. J Cardiopulm Rehabil Prev 2011 Dec 21. [Epub ahead of print]
5. Nolte E, Knai C, Hofmarcher M, Conklin A, Erler A, Elissen A, Flamm M, Fullerton B, Sönnichsen A, Vrijhoef HJ. Overcoming fragmentation in health care: chronic care in Austria, Germany and The Netherlands. Health Econ Policy Law 2012;7(1):125-46.
6. Fullerton B, Erler A, Pohlmann B, Gerlach FM. Predictors of dropout in the German disease management program for type 2 diabetes. BMC Health Serv Res 2012;12:8.
Furthermore, DISEMEVAL partners have led on and/or contributed to technical reports that are available to the public on the projects website (http://www.dismeval.eu/publications/ online for further details). These reports are:
1. Conklin A, Nolte E. Disease management evaluation. A comprehensive review of current state of the art. Santa Monica: RAND Corporation, 2011
2. Nolte E, Conklin A, Adams J, Brunn M, Cadier B, Chevreul K, Durand-Zaleski I, Elissen A, Erler A, Flamm M, Frølich A, Fullerton B, Jacobsen R, Knai C, Krohn R, Pöhlmann B, Saz Parkinson Z, Sarria Santamera A, Sönnichsen A, Vrijhoef H. Evaluating chronic disease management. Recommendations for funders and users. RAND Europe/DISMEVAL Consortium, 2012.
3. Nolte E, Hinrichs S, eds. DISMEVAL. Developing and validating disease management evaluation methods for European healthcare systems. Final report. RAND Europe/DISMEVAL Consortium, 2012.
Dissemination is expected to be ongoing beyond the formal project period with a series of consortium publications underway/close to submission including a volume to be published as a monograph in summer 2012, reporting on 'Chronic disease management in Europe - An overview of 13 countries'.
List of Websites:
http://www.dismeval.eu
Aims and objectives
DISMEVAL aimed to contribute to developing new research methods and to generating the evidence base to inform decision making in the field of chronic disease management evaluation.
Specifically, its three-year programme of work sought to:
(1) review current approaches to chronic care management and their evaluations, as implemented by EU Member States at national and regional level,
(2) explore the policy context for chronic disease management in European countries,
(3) develop and validate evaluation methods for chronic disease management using data from existing programmes and approaches, and
(4) formulate recommendations for scientifically sound yet operationally feasible evaluation approaches for chronic disease management that are relevant to policies at EU and the wider European level and internationally.
Main findings
The DISMEVAL project has achieved these aims at three levels. First, we inform on the range of approaches to chronic care adopted in 13 European countries, and, by reflecting on facilitators and barriers to implementation, we provide policy makers and practitioners with a portfolio of options to advance care in a given policy context. We note how there remain considerable challenges towards the development of a policy framework for a strategic response to chronic disease, with considerable variation in the nature and scope of approaches to care across Europe. Challenges include a continued focus on complications management, with some movement towards more systematic disease management, and an overall lack of coordination between levels; failure to integrate risk minimisation and disease prevention with other components along the care spectrum; misalignment of financial incentives that tend to reward 'cure' over prevention; and a disjoint between intent, at national level, to enhance coordination and integration, and ability at regional or local level to translate these ambitions into practice. These observations emphasise the need for the development of a coherent response to chronic disease that takes account of the various tiers in the system and along the care continuum, with involvement of professionals forming a crucial component for achieving sustainable change.
Second, through testing and validating different evaluation methods using existing data we advance the research base in evaluation design and methodology, inform the design of future evaluations and enhance their value for decision-making. While acknowledging that a randomised controlled design should generally be considered as the most robust way of determining the effectiveness of a given intervention, the DISMEVAL project has identified and tested a wide range of methods for the evaluation of disease management that can be employed in situations where randomisation is not possible. We emphasise that rigorous evaluation is still possible even where baseline or predefined control groups are not available and how advanced designs can help better understand how different care components and processes might be effective for managing chronic disease in patients with different characteristics. Future evaluation work drawing on such approaches can provide insight into what works for who in disease management, a question that randomised trials have thus far been unable to answer. Project work further highlighted, through the introduction of statistical controls for selection or statistical matching, how findings were substantially different from simple comparisons of patients receiving a given disease management intervention and those who do not. Thus, the DISMEVAL project has shown how the use of randomisation or other methods of control is necessary to accurately assess the impact of such interventions. It also identified a range of methods that can be employed successfully to implement such controls.
Third, through the development of a report on recommendations for the evaluation of chronic disease management interventions DISMEVAL contributes to strengthening evidence-based policy and practice in Europe. The report is specifically targeted at policymakers, programme operators and researchers, explaining choices, options and trade-offs at methodological, practical and wider societal levels.
Challenges and lessons learned
Overall, evidence presented by DISMEVAL confirms the substantial heterogeneity across disease management interventions. This highlights how a given evaluation needs to be not only summative, but also include formative components to understand how interventions and programmes can be improved. Such consideration is particularly important against the background that the implementation of disease management is essentially a process of social change, it is important to combine quantitative data on effects with qualitative information concerning contexts. Use of mixed methods can ensure that disease management evaluation provides insight into how specific local conditions influence the outcomes of a given programme.
Work undertaken within DISMEVAL on evaluation metrics and methods was limited to disease-specific programmes, mirroring much of the existing research evidence that has focused on the management of a few specific diseases, such as diabetes. Interventions studied here tended to target those with good baseline status on key measures. There appeared to be little improvement on key scores for these baseline patients, and measures based on average improvements in scores tend to average those patients among whom improvement is possible with those among whom further improvement should not be expected. There is a need for further serious consideration to the identification of appropriate measures of success for disease management interventions that takes account of these differential impacts. Importantly however, there has been less focus on individuals with coexisting conditions or multiple health problems, even though it is this rapidly increasing population, with multiple disease processes and with diverse and sometimes contradictory needs, who pose the greatest challenge to health systems. Furthermore, as we have shown, the impact of chronic disease management interventions will depend, to a considerable extent, on the specific features of the healthcare setting within which they are introduced, and this observation seems to hold both within and between care systems. However, this work has shown how it can be possible to learn from the experiences of others.
Project Context and Objectives:
The rising burden of chronic disease presents challenges for all health systems.
[1] In the European Union, in 2006, between 20 and over 40 percent of the population aged 15 years and over reported a long-standing health problem and one in four currently receives long-term medical treatment.
[2] Other studies find the prevalence of common chronic disorders to be around 50 percent among adults aged 18 and older in seven high-income countries, including Germany, the Netherlands and the UK.
[3] Assessing the precise level, distribution and nature of the chronic disease burden in Europe remains a challenge;
[4]yet, it is clear that chronic diseases are an important cause of premature mortality and disability, greatly impacting on the years of life lived in good health. In high-income countries, depressive disorder, ischaemic heart disease and cerebrovascular disease, dementia, chronic obstructive pulmonary disease (COPD) and diabetes are among the ten leading contributors to the burden of disease
[5] with diabetes projected to rise further in importance during the next two decades, especially against the background of increasing levels of overweight and obesity.
[6]Chronic diseases pose a sizeable burden for national economies, with some studies estimating the associated costs at up to seven percent of a country's gross domestic product.
[7] Societal costs arise partly as a result of direct healthcare costs, including from healthcare utilisation, medication and potentially costly interventions, but these can also be caused by other factors, such as a decrease in work productivity.
[8] These challenges add to the complexity facing health systems, which require effective measures to prevent disease through reducing the major chronic disease risk factors and addressing influences that drive exposure
[9], while also providing services to meet the requirements caused by chronic health problems, ensuring that people with established illnesses can participate in society.
The DISMEVAL project
DISMEVAL (Developing and validating Disease Management Evaluation methods for European healthcare systems), a European collaborative project, aimed to contribute to developing new research methods, and to generate the evidence base to inform decision-making, in the field of chronic disease management evaluation.
The DISMEVAL project sought:
(i) to enhance our understanding of the use of various approaches to the evaluation of disease management in Europe and to identify examples of best practice and lessons learned;
(ii) to provide evidence-based recommendations to policymakers, programme operators and researchers on which evaluation approach is most useful in a given context;
(iii) to promote and support the networking and coordination of research and innovation activities relating to scientific knowledge and policy development in this area, building on existing work carried out in Member States and at the wider European level; and
(iv) to analyse scientific knowledge and developments as well as actions and policies within EU Member States, develop tools to assist policy analysis, and work in close collaboration with the Commission services, networks and experts in this area, and with stakeholder groups and various agencies to provide scientific input to support ongoing and planned actions and policies in the European Union.
The DISMEVAL project spanned multiple scientific domains by bringing together ten partners in seven European countries, representing a variety of disciplines including evaluation science, chronic care, disease management design and operations, epidemiology, economics, and health policy. Partner countries were selected to represent the variety of chronic disease management interventions that can be found across the EU, stretching from large-scale, population-based programmes to smaller, provider-centric interventions.
Within this broad context, DISMEVAL's three-year programme of work was developed around a set of key objectives:
(i) to review current approaches to chronic care implemented by EU Member States at national and regional level and to examine whether and how these interventions are being evaluated;
(ii) to enhance our understanding of the impact of macro-level health system features on chronic care interventions through exploring the policy context for chronic disease management in European countries;
(iii) to develop and validate methods for evaluating disease management programmes using data from existing programmes and approaches by employing a range of evaluation designs and assessing the sensitivity of findings to selected methods; and
(iv) to formulate recommendations for scientifically sound yet operationally feasible evaluation approaches for chronic disease management that are relevant to planned and ongoing policies at the EU and wider European level as well as internationally.
This was to be achieved through a programme of work that can be differentiated into three phases:
(1) review of current approaches to the implementation and evaluation of chronic disease management;
(2) testing and validation of methods and metrics for the evaluation of chronic disease management; and
(3) development and recommendations for methods and metrics for the evaluation of chronic disease management. Here we briefly outline each of the three phases and the work packages carried out therein.
Phase 1: Review of current approaches to the implementation and evaluation of chronic disease management in Europe
Phase 1 comprised work packages two to four. Work package 2 (WP2) 'Approaches to chronic disease management' sought to review approaches to managing chronic conditions that have been developed and/or implemented in different countries in Europe, and to assess whether and how countries evaluate approaches to chronic disease management, so feeding into work package 3 (WP3) 'Assessment of disease management evaluation approaches in Europe'. Thus, based on data collected in WP2, WP3 aimed to provide an overview of the types of evaluation approaches that are being used in Europe to assess the impact of structured approaches to disease management on the cost and quality of chronic illness care. Both work packages informed WP4 'Assessing the policy context for chronic disease management in Europe', which aimed to assess the overall policy framework for chronic disease management in selected European countries.
Phase 2: Testing and validation of methods and metrics for the evaluation of chronic disease management
Phase 2 of the project included work packages 5-10. The main aim of this phase was to utilise data from existing chronic disease management programmes, or their equivalent, in partner countries, to test and validate different evaluation options reviewed in Phase 1 of the project. The countries included were Austria (WP5), Denmark (WP6), Germany (WP7), France (WP8), the Netherlands (WP9), and Spain (WP10).
Phase 3: Development and recommendations for methods and metrics for the evaluation of chronic disease management
Phase 3 comprised work package 11, which sought to summarise the findings and to present best practice and lessons learned from work undertaken in WPs 5-10. It further aimed to present validated recommendations on performance indicators and evaluation methods for disease management programmes or their equivalent. The final output of this work package is a report on evaluation of chronic disease management that outlines choices, options and trade-offs to policymakers, programme operators and researchers, and with recommendations presented alongside their rationale.
References
[1] Yach D, Hawkes C, Gould C, Hofman K. The global burden of chronic diseases. JAMA 2004;291:2616-22.
[2] TNS Opinion and Social. Health in the European Union. Special Eurobarometer 272e. Brussels: European Commission, 2007.
[3] Schoen C, Osborn R, Doty M, Bishop M, Peugh J, Murukutla N. Towards higher-performance health systems: adults' health care experiences in seven countries. Health Affairs 2007;26:w717-w34.
[4] Pomerleau J, Knai C, Nolte E. The burden of chronic disease in Europe. In: Nolte E, McKee M, editors. Caring for people with chronic conditions. A health system perspective. Maidenhead: Open University Press, 2009:15-42.
[5] World Health Organization. The global burden of disease: 2004 update. Geneva: World Health Organization, 2008.
[6] Finucane M, Stevens G, Cowan MD, G, Lin J, Paciorek C, Singh G, et al. National, regional, and global trends in body-mass index since 1980: systematic analysis of health examination surveys and epidemiological studies with 960 country-years and 9.1 million participants. Lancet 2011;377:557-67.
[7] Suhrcke M, Nugent R, Stuckler D, Rocco L. Chronic disease: an economic perspective. Oxford: Oxford Health Alliance, 2006.
[8] Suhrcke M, Fahey D, McKee M. Economic aspects of chronic disease and chronic disease management. In: Nolte E, McKee M, editors. Caring for people with chronic conditions: A health system perspective. Maidenhead: Open University Press, 2008:43-63.
[9] Novotny T. Preventing chronic disease: everybody's business. In: Nolte E, McKee M, editors. Caring for people with chronic conditions: A health system perspective. Maidenhead: Open University Press, 2008:92-115.
[10] Mattke S, Seid M, Ma S. Evidence for the effect of disease management: Is $1 billion a year a good investment? Am J Manag Care 2007;13:670-676.
Project Results:
This section presents the main findings of work carried out within the DISMEVAL project. It first reviews current approaches to the implementation and evaluation of chronic disease management. Second, it reports on the testing and validation of methods and metrics for the evaluation of chronic disease management. The third stage involved the development and recommendations for methods and metrics for the evaluation of chronic disease management; we here reflect on some of the challenges identified, which may be deemed necessary to consider for users, funders and researchers interested in the evaluation of structured approaches to chronic disease management. It concludes with identifying continued challenges and lessons learned. A more detailed account of the findings of the DISMEVAL project is documented in the Final Report, with an accompanying volume providing recommendations for funders and users on methods and metrics for the evaluation of chronic disease management. Both reports are available at http://www.dismeval.eu/publications/.
Reviewing approaches to chronic disease management in Europe
The review of approaches to chronic disease management in Europe drew on a common template for data collection, which was based on a structured questionnaire used in the framework of a previous study by Nolte, Knai and McKee (2008)[1], and informed, to a great extent, by the Chronic Care Model (CCM) developed by Wagner and colleagues in the United States.[2] Accordingly, the template sought to gather information on:
(i) the health system and policy context;
(ii) the type and format of approaches to managing chronic disease;
(iii) the evaluation of existing approaches; and
(iv) system markers of success or failure for organisational approaches to chronic disease management.
Selection of countries for review was guided by three main criteria in order to capture the range of approaches to funding and governing health care across Europe; the range of stages in economic development; and geographical spread across the EU. On this basis, we selected 13 countries for detailed analyses: Austria, Denmark, England, Estonia, France, Germany, Hungary, Italy, Latvia, Lithuania, the Netherlands, Spain and Switzerland (the only non-EU country). Principal data collection was carried out by key informants in each country over the period of June 2009 to December 2009, with two rounds of updates in autumn 2010 and summer 2011.
Based on this survey of approaches to chronic disease management in 13 European systems we found that most countries have sought to create a regulatory and policy framework to respond to chronic disease during recent years. These generally aim to promote approaches that better integrate care and improve coordination between sectors and levels of care but countries differ with regard to their vision towards controlling and managing chronic disease. Likewise, approaches to chronic disease management vary in scope and nature across Europe.
While our review did not attempt to present a comprehensive inventory of all approaches being implemented in a given country, some of the key observations were that:
- the majority of countries tend to focus on care models for populations with defined conditions, most frequently diabetes type 2, and involve some form of GP-led care coordination
- nurse-led approaches are becoming more common although there are differences in the degree to which nurses can operate independently
- patient access is typically granted in line with access to usual care, although many approaches are being implemented in selected geographical regions so potentially limiting access to defined population groups
- the majority provide some form of patient self-management support, although the level and scope of support offered varies
- the overall the use of clinical information systems for chronic disease management tends to be the least developed strategy in most approaches
Based on the survey data collected within DISMEVAL, we also examined whether and how chronic disease management models and programmes, or their equivalent that have been or are being implemented in European countries, are being evaluated. We find that while most initiatives reviewed had undergone some form of evaluation or had evaluation plans in place, the nature and scope of evaluations varied, with differences in objectives, design, the performance metrics used, the length of observation and approaches to data collection. We identified a range of challenges posed to the more systematic use of evaluation of complex healthcare interventions such as disease management in European health systems, including a perceived lack of an evaluation culture in some settings, alongside lack of financial and human resources to conduct systematic evaluation. We note that evaluation of approaches to disease management reviewed would likely benefit analytically from increased use of sophisticated statistical techniques, but also conceptually from drawing more explicitly on one of the many possible theories of behaviour change to better link the choice of performance measures to the goals of the intervention (and of the evaluation). There may also be scope to more systematically draw in mixed-methods approaches to help place observed quantitative findings into the context within which the intervention under evaluation is embedded. More information is needed about the characteristics of the intervention and its intended populations, requiring greater specification of the wider context so as to improve comparisons and potential transferability across settings and countries in Europe.
In order to place the survey findings reviewed in the preceding sections into the wider context, we carried out interviews with key informants in a sample of countries that formed the core group of the DISMEVAL project (Austria, Denmark, France, Germany, the Netherlands and Spain). This provided an opportunity to learn how representatives from different sectors are approaching chronic care, and their perception of the policy framework for providing a strategic response to chronic disease. Some of the reported challenges included a continued focus of chronic care on complications management, with some movement towards more systematic disease management, and an overall lack of coordination between levels; failure to integrate risk minimisation and disease prevention with other components along the care spectrum; misalignment of financial incentives that tend to reward 'cure' over prevention; and a disjoint between intent, at national level, to enhance coordination and integration, and ability at regional or local level to translate these ambitions into practice. While some of these observations are perhaps not specific to chronic care as such, they emphasise the need for the development of a coherent response to chronic disease that takes account of the various tiers in the system and along the care continuum, with involvement of professionals, who exert a large degree of control in healthcare organisations such as primary care practices and hospitals. As we and others have argued elsewhere, failure to engage them in the reform process is likely to hamper sustainable change.[3-5]
Testing and validation of methods and metrics for the evaluation of chronic disease management
The testing and validation of methods and metrics for the evaluation of chronic disease management comprised studies carried out in six countries and using data from existing interventions. These were: disease management programmes for diabetes type 2 in Austria and Germany, diabetes care groups in the Netherlands, provider networks for diabetes and for cancer in France, an interdisciplinary and sectoral rehabilitation programme for people with chronic obstructive pulmonary disease and for diabetes in Denmark, and a nurse-led intervention targeting a working-age population at risk of cardiovascular disease in Spain. All interventions were implemented in a non-experimental setting; the only exception was the diabetes disease management programme in Salzburg, Austria, which was implemented as a pragmatic cluster-randomised controlled study.
As interventions and the setting in which they were implemented varied, so did their approaches to testing and validation approaches to evaluation. Thus, country studies aimed to:
- quantify differences in effect sizes of structured care within a diabetes disease management programme using randomised and non-randomised controlled and non-experimental designs (Austria)
- test different approaches to identify treatment-control matches in non-experimental settings and quantify the likely impact on the effect estimate of an interdisciplinary and sectoral intervention for patients with chronic obstructive pulmonary disease or diabetes (Denmark)
- compare different methods to adjust for confounding in a non-experimental setting using routine data to assess intervention effect of a diabetes disease management programme (Germany)
- test for selection bias for participating in a structured care programme for diabetes (Germany, France )
- employ advanced methods of disease management evaluation in non-experimental settings to better understand differential effects of structured care components on subpopulations (Netherlands, Spain).
Secondary goals of each analysis further included evaluating the effects of the intervention under study on patient outcomes more generally (all countries), alongside better understanding of the usefulness of current approaches to evaluation in the context of intervention practice (France, Netherlands) and to derive recommendations for further development of interventions and evaluation practice. We here provide an overview of the main findings of country reports.
Evaluating the diabetes disease management programme in Austria using an uncontrolled design overestimates treatment effects
In Austria, the evaluation of the effect of the diabetes disease management programme 'Therapie Aktiv', using a randomised controlled design, found a reduction in HbA1c levels of 0.13 percent after one year. This effect was not statistically significant while measures of process quality such as regular eye and foot examination and patient education improved significantly. In contrast, using an uncontrolled before-after design, treatment effect was estimated at a significant reduction in HbA1c levels of 0.41 percent in the intervention group. Extrapolating these findings to clinically relevant endpoints such as number needed to treat to avoid one case of myocardial infarction or one diabetes complication over a period of ten years, the uncontrolled before-after design overestimated the treatment effect by a factor of three. These findings support the general notion that use of a randomised controlled design should be considered as the main means for evaluating treatment effect of a structured care intervention. Cluster randomisation as applied in the case of Austria can be seen to provide a pragmatic approach to DMP evaluation where a randomised controlled design is not feasible.
Different approaches to identify treatment-control matches in a non-experimental intersectoral intervention for patients with chronic obstructive pulmonary disease in Denmark provide different estimates of treatment effect
In Denmark, the evaluation of the effect of an interdisciplinary and intersectoral rehabilitation programme for patients with chronic obstructive pulmonary disease (COPD) found effect sizes to vary with the method of constructing control groups for the intervention-control and the difference-in-difference analysis. For example, propensity score matched sampling lowered the magnitude of the predicted intervention effect for COPD-specific hospital bed days when compared with controls created by random sampling. Likewise, control groups not matched by disease severity overestimated the effects of pulmonary rehabilitation on COPD-specific hospital contacts and bed days.
The study considered the method of control group construction and matching using propensity scoring and the use of difference-in-difference analyses to assess intervention effect to be optimal to evaluate the impact of interventions in a non-experimental setting. As for the impact of the overall programme, the case study provided evidence that the intervention might have decreased the pace of disease progression in the intervention group, which was reflected in a non-significant increase in COPD hospital contacts, bed days, ambulatory visits and emergency room visits in the intervention group while these indicators significantly increased in the entire sample.
Different methods to adjust for baseline differences in a non-experimental setting using routine data to assess intervention effect of a diabetes disease management programme in Germany result in similar effect measures
In Germany, the evaluation of intervention effect of a diabetes disease management programme (DMP) in a non-experimental setting found that different matching and/or weighting methods used resulted in similar effect measures for the outcome variables analysed. It highlighted how the applicability of available statistical methods and tools to perform a sound evaluation of programme effects is conditional on the availability of reliable baseline data prior to enrolment in the intervention, to enable adjustment of differences between intervention and control group. Therefore, a great effort should be made to collect detailed and valid data to maximise the usefulness of routine data for evaluation.
As for the impact of the overall programme, the case study confirmed the findings of other studies that participation in the diabetes DMP improved process parameters in diabetes care especially those related to the monitoring of the disease. However, intensified care in the programme was accompanied by higher overall costs, primarily because of higher prescription costs. In order to draw valid conclusions about DMP effects on clinical endpoints such as mortality or micro- and macrovascular complications, a longer study period consistent with what is known about the time course of the disease should be chosen.
Patient selection for participating in a structured care programme for diabetes in Germany and France leads to over- and/or underestimation of findings of effect of the intervention
The evaluation of the diabetes disease management programme in Germany also observed a significant reduction of mortality in the intervention group. This effect was likely attributable to GPs systematically excluding patients from joining the programme who were more likely to die in the near future. Adjusting for baseline variables reduced this effect compared with the unadjusted analysis. Future analyses should include further adjustment for variables suited to predict short-term mortality risk. Furthermore, a longer observation period would be required to assess whether the mortality difference diminishes over the years.
Evidence of patient selection into the intervention was also observed in the case study for France, which examined patient characteristics of those enrolled with a diabetes provider network. These patients were found to be of younger age whose diabetes was diagnosed more recently but showed evidence of worse glycaemic control than patients in the reference population. By comparing a standard uncontrolled pre-post evaluation design with a pre-post design after calibration with the reference population at baseline, the analysis further showed that the uncontrolled evaluation design overestimated intervention effect on a number of clinical outcomes, including improvements in HbA1c levels and body mass index while underestimating deterioration in renal function in diabetic patients.
Advanced methods of disease management evaluation in non-experimental settings in the Netherlands and Spain help understand the differential effects of care components on subpopulations and so inform further development of structured care approaches
Given that most randomised controlled trials are conducted in academic settings and provide limited insight into the impact of disease management in the everyday practice of health care, in the Netherlands, the evaluation of integrally financed, regional disease management programmes for diabetes used a variety of approaches to better understand the effects of different programmes. It found that applying methods that permit for subgroup analyses provide important new insights into differential component effects. Thus, while the overall analysis of intervention effects found only modest impacts of the care programmes on the health status of patients with diabetes type 2, subgroup analyses revealed disease management to be considerably more effective for patients with poor baseline clinical values. As the vast majority of patients included in the analyses had good baseline values of most clinical endpoints, this differentiation provided a plausible explanation for the modest overall effects of the intervention. This suggests that further development of the intervention should move towards a more tailored approach to diabetes care, in which the characteristics of patients directly determine the processes of diabetes care, including self-management support. Such a move will, however, require improvements in the current systems for data registration to provide valid and reliable information on patients' health status to determine care intensity.
Similarly, in assessing the effect of a nurse-led intervention targeting a working-age population at risk of cardiovascular disease in Spain, the use of advanced methodological approaches that permit for subgroup analysis in a non-experimental setting provided important new insights into the effects of the intervention.
Taken together, observations emerging from the six country reports suggest that in non-experimental settings, the creation of statistical controls or adjustment using propensity or matching methods was feasible using routine data sets. Application of such methods consistently provided different estimates, generally smaller, of intervention effectiveness. While the literature suggests that more aggressive methods such as boosted regression may provide a more suitable means for matching, results from studies presented here that compared alternative matching methods found results to be similar. Analyses presented here also provided insights into programme effectiveness without introducing an external control by examining variation across interventions such as in the Netherlands, or by excluding patient populations close to the inclusion threshold (Spain).
Practical considerations for evaluation
Data availability constitutes a considerable challenge to evaluation research, even where evaluation design and capacity is of high quality. Where routinely collected data is used, this may be incomplete and may require systematic scrutiny to assess the implications of missing data for the analysis and interpretation of findings; this may require additional resources where missing data has to be imputed from elsewhere. Even where routine data is fairly complete, it may be inadequate for the purpose of evaluation as it is typically used for administrative purposes only and may not record outcomes of interest. Evaluations may therefore necessitate new data collection, with consequent resource implications.
Evidence of work carried out within the DISMEVAL project has highlighted the many opportunities and challenges arising from the use of routinely collected data for the evaluation of disease management. Routine data have the advantage of relatively easy access and data are available for a large patient sample, capturing information on daily practice of healthcare provision and permitting assessment of relatively long-term outcomes, such as mortality within a limited timeframe. Furthermore, financial costs of data collection tend to be low. These advantages potentially outweigh the challenges associated with the considerable volume of missing data that tends to be common, in particular for data collection systems that are newly introduced and/or for which validity checks are not routinely carried out, as has been observed for the German case study which drew on routinely collected data from a statutory health insurance fund. Therefore, any evaluation that makes use of routine data systems will have to allow sufficient time to enable validity checks and a good understanding of the context within which data collection takes place, in particular in relation to the range of outcome variables to be considered for evaluation where data is not collected for research but for administrative purposes. Experiences from the Dutch and the Danish case studies stressed that where new data documentation on disease management interventions is being established, this will require supportive infrastructure for data to be usefully utilised for evaluation. The example from France has illustrated how heterogeneity of routine data collection systems can pose challenges for their use in evaluation, highlighting the need for greater collaboration between those responsible for data collection.
The experiences of DISMEVAL case studies further suggest that even where routine data can be accessed in principle, full exploitation of data available may be compromised, which had been the case for data from statutory insurance funds in Germany and Austria. While maintaining data confidentiality is a serious concern in any research, there may be a need for decision-makers to consider putting safeguards in place to allow for full access of routinely collected data sets while maintaining high standards of data privacy and confidentiality, such as is illustrated in the Danish case study. An important lesson drawn from all case studies presented in DISMEVAL is the necessity, in the evaluation plan, to allow for sufficient time to negotiate access to data and, where necessary, to set sufficient additional resources aside to allow for data extraction by the holder of the data.
Challenges and lessons learned
The DISMEVAL project set out to enhance our understanding of the use of various approaches to the evaluation of disease management in Europe, to identify examples of best practice and lessons learned and to provide evidence-based recommendations to policymakers, programme operators and researchers on evaluation approaches that may be most useful in a given context.
The work carried out within DISMEVAL set out on the premise that experimental research designs, particularly randomised controlled trials, are generally considered to be the most rigorous way of determining the effectiveness of a given intervention.[6] The Austrian case study in DISMEVAL described above has illustrated how it can be feasible to employ a randomised design in routine settings where the context allows for such a design to be applied. However, use of an experimental design may not be possible (or desirable) for population-wide disease management interventions, which are frequently implemented in an operational setting and do not usually have a control group available, such as in Germany and the Netherlands. For example, in the Dutch case study use of an experimental approach was not possible due to the nation-wide roll-out of structured care approaches for diabetes and the unsuitability of using historic controls. Furthermore, although randomised studies are generally considered to form the most rigorous means to assess intervention effect, the scientific rigour of required designs limits the generalisability of findings to larger and inherently more heterogeneous populations of, for example, chronically ill patients. Selection bias poses a threat to randomised designs as it does for non-randomised designs as we have highlighted.
Observational study designs are more suitable for 'real-world' disease management evaluations, keeping their methodological limitations in mind. Given that disease management is essentially a population-based care strategy, advancing observational study designs is crucial to come to strong conclusions on how to best target subgroups of chronically ill patients in the daily healthcare practice. The DISMEVAL project has identified and tested a wide range of methods that can be employed in situations where randomisation is not possible, emphasising that rigorous evaluation is still possible even where baseline or predefined control groups are not available. Project work further highlighted, through the introduction of statistical controls for selection or statistical matching, how findings were substantially different from simple comparisons of patients receiving a given disease management intervention and those who do not. Thus, the DISMEVAL project has shown how the use of randomisation or other methods of control is necessary to accurately assess the impact of such interventions. It also identified a range of methods that can be employed successfully to implement such controls.
Different (combinations of) care components and processes might be effective for managing chronic disease in patients with varying age, disease duration, health status, co-morbidity, education level, socio-economic status and so on. Contrary to most methods aimed at evaluating disease management interventions, which focus on assessing a single treatment effect, meta-analysis and meta-regression allow for investigations of which patient groups will benefit most from which treatment. Therefore future evaluation work drawing on such approaches can provide insight into what works for whom in the area of disease management, a question that randomised trials thus far have been unable to answer.[7] In addition, meta-regression analyses can be adjusted for baseline (prognostic) factors, which can increase the power of the analysis to detect a true treatment effect and allows adjustment for confounding factors, which is a particular advantage for analyses of observational data.
Overall, evidence presented by DISMEVAL confirms the substantial heterogeneity across disease management interventions. This highlights how a given evaluation needs to be not only summative, but also include formative components to understand how interventions and programmes can be improved. Such consideration is particularly important against the background that the implementation of disease management is essentially a process of social change, it is important to combine quantitative data on effects with qualitative information concerning contexts. Use of mixed methods can ensure that disease management evaluation provides insight into how specific local conditions influence the outcomes of a given programme.
Work undertaken within DISMEVAL on evaluation metrics and methods was limited to disease-specific programmes, mirroring much of the existing research evidence that has focused on the management of a few specific diseases, such as diabetes. Interventions studied here tended to target those with good baseline status on key measures. There appeared to be little improvement on key scores for these baseline patients, and measures based on average improvements in scores tend to average those patients among whom improvement is possible with those among whom further improvement should not be expected. There is a need for further serious consideration to the identification of appropriate measures of success for disease management interventions that takes account of these differential impacts. Importantly however, there has been less focus on individuals with coexisting conditions or multiple health problems[8-9], even though it is this rapidly increasing population, with multiple disease processes and with diverse and sometimes contradictory needs, who pose the greatest challenge to health systems.[10] Furthermore, as we have shown, the impact of chronic disease management interventions will depend, to a considerable extent, on the specific features of the healthcare setting within which they are introduced, and this observation seems to hold both within and between care systems. However, this work has shown how it can be possible to learn from the experiences of others.
References
[1] Nolte E, Knai C, McKee M, editors. Managing chronic conditions: Experience in eight countries. Copenhagen: World Health Organization on behalf of the European Observatory on Health Systems and Policies, 2008.
[2] Wagner E. Chronic disease management: What will it take to improve care for chronic illness? Effective Clinical Practice 1998;1:2-4.
[3] Nolte E, McKee M. Making it happen. In: Nolte E, McKee M, editors. Caring for people with chronic conditions: A health system perspective. Maidenhead: Open University Press, 2008:222-244.
[4] Ham C. Improving the performance of health services: the role of clinical leadership. Lancet 2003;361:1978-1980.
[5] Nolte E, Knai C, Hofmarcher M, Conklin A, Erler A, Elissen A, Flamm M, Fullerton B, Sonnichsen A, Vrijhoef HJM. Overcoming fragmentation in health care: chronic care in Austria, Germany and the Netherlands. Health Econ Polit Law 2012;7:125-146.
[6] Conklin A, Nolte E. Disease management evaluation. A comprehensive review of current state of the art. Santa Monica: RAND Corporation, 2011.
[7] Shojania K, Ranji S, McDonald K, Grimshaw J, Sundaram V, Rushakoff R, et al. Effects of quality improvement strategies for type 2 diabetes: a meta-regression analysis. JAMA 2006;296:427-40.
[8] Piette J, Kerr E. The impact of comorbid chronic conditions on diabetes care. Diabetes Care 2006;29:725-31.
[9] Struijs J, Baan C, Schellevis F, Westert G, van den Bos G. Comorbidity in patients with diabetes mellitus: impact on medical health care utilization. BMC Health Serv Res 2006;6:84.
[10] Nolte E, McKee M. Caring for people with chronic conditions. A health system perspective. Copenhagen: World Health Organization on behalf of the European Observatory on Health Systems and Policies, 2008.
Potential Impact:
The DISMEVAL project contributes to advancing the evidence base and informing decision making in addressing chronic illness in Europe at three levels. First, we inform on the range of approaches to chronic care adopted in 13 European countries, reflecting on facilitators and barriers to implementation. It provides policy makers and practitioners with a portfolio of options to advance chronic care approaches in a given policy context. We note how that there remain considerable challenges towards the development of a policy framework for providing a strategic response to chronic disease, with considerable variation in the nature and scope of approaches to chronic disease management across Europe. Challenges included a continued focus of chronic care on complications management, with some movement towards more systematic disease management, and an overall lack of coordination between levels; failure to integrate risk minimisation and disease prevention with other components along the care spectrum; misalignment of financial incentives that tend to reward 'cure' over prevention; and a disjoint between intent, at national level, to enhance coordination and integration, and ability at regional or local level to translate these ambitions into practice. These observations emphasise the need for the development of a coherent response to chronic disease that takes account of the various tiers in the system and along the care continuum, with involvement of professionals forming a crucial component for achieving sustainable change.
Second, through testing and validating different evaluation methods using existing data we advance the research base in evaluation design and methodology, inform the design of future evaluations and enhance their value for decision-making. Thus, while acknowledging that a randomised controlled design should generally be considered as the most robust way of determining the effectiveness of a given intervention, the DISMEVAL project has identified and tested a wide range of methods for the evaluation of disease management interventions that can be employed in situations where randomisation is not possible. We emphasise that rigorous evaluation is still possible even when baseline or predefined control groups are not available, and how advanced designs can help better understand how different care components and processes might be effective for managing chronic disease in patients with different characteristics. Future evaluation work drawing on such approaches can provide insight into what works for who in the area of disease management, a question that randomised trials have thus far been unable to answer. Project work further highlighted, through the introduction of statistical controls for selection or statistical matching, how findings were substantially different from simple comparisons of patients receiving a given disease management intervention and those who do not. Thus, the DISMEVAL project has shown how the use of randomisation or other methods of control is necessary to accurately assess the impact of such interventions. It also identified a range of methods that can be employed successfully to implement such controls.
Third, through the development of a report on recommendations for the evaluation of chronic disease management interventions, it contributes to strengthening evidence-based policy and practice in Europe. The report is based on the premise that policymakers, programme operators and researchers need validated approaches for the evaluation of disease management to design effective, efficient and equitable interventions to improve the care for people with chronic conditions.
Of particular relevance there are two components of evaluation:
(1) the performance indicators that are used to capture the impact of a given programme and
(2) the attribution strategy, which enables identifying any observed changes in selected indicators as programme effect rather than as a consequence of other factors, such as secular trends or changes in the treatment (ie establishing the counterfactual - whether the change could have been observed in the absence of the programme).
These methods need to be scientifically sound but also operationally feasible to address common threats to validity in disease management evaluation.
The report is therefore targeted at policymakers, programme operators and researchers, explaining choices, options and trade-offs for the evaluation of disease management based on analyses undertaken within the DISEMEVAL project. At the outset it is important to note that many of the issues discussed in this report are not specific to disease management evaluation but can be seen to apply to any evaluation of complex interventions in healthcare.
However, there are specific concerns around evaluation design and metrics that are of relevance to disease management evaluation and which are given particular attention, including:
(i) the context for evaluating disease management, exploring the reasons for undertaking evaluation in the first place and explaining some of the underlying principles for doing so;
(ii) the methods and metrics of disease management evaluation, focusing specifically on themes that have emerged as being pertinent to work carried out within the DISMEVAL project;
(iii) practical considerations for disease management evaluation, based on experience of work undertaken in DISMEVAL; and
(v) the broader challenges and lessons learned that may be relevant for policymakers, funders and practitioners interested in the use and usefulness of disease management evaluation more generally.
We therefore expect this publication to provide a major resource to guide the evaluation of disease management interventions in European settings and so contribute to strengthening the evidence-base required to inform the selection of efficient and effective interventions to address the growing burden of chronic disease in Europe.
Beyond this report which is, as noted above, specifically targeted at policymakers, programme operators and practitioners, work carried out within the DISMEVAL project has been disseminated widely, both at national and international levels. Progress on work and key findings were primarily communicated through presentations in the context of meetings and conferences targeting researchers, policy makers and practitioners in Europe, the Americas, India and Australia; with peer-reviewed publications becoming a more important source for communication especially in the third and final year of the project. We expect dissemination of project findings to continue beyond the formal completion of the DISMEVAL project, both through presentations and discussions at (inter)national conferences and meetings and, in particular, publications, both in the academic literature as well as in the form of policy briefs and monographs targeting the policy and practitioner community in particular.
Key dissemination achievements during the lifetime of the DISMEVAL project included:
1. The organisation and presentation of DISMEVAL workshops at European conferences:
- Presentation of the DISMEVAL project at the Working conference Health Services Research in Europe, the Hague, the Netherlands, 8-9 April 2010. This conference provided a platform to present the project to a wider audience of health services researchers and policy makers in Europe
- A workshop 'Improving chronic illness care: experiences with different approaches internationally' at the 27th ISQua International Conference, Paris, France, 10 October 2010. This workshop sought to discuss current evidence of how disease management and other innovations impact chronic conditions and how health system characteristics influence the impact of such interventions. It involved presentations by three DISMEVAL partners (RAND, GUF, UM)
- A DISMEVAL workshop organised by the DISMEVAL Consortium at the 3rd European Public Health Conference 'Integrated Public Health', Amsterdam, the Netherlands, 11 November 2010. The workshop aimed to provide a forum for discussion of conceptual, methodological and policy-related challenges of the evaluation of approaches targeting people with chronic conditions, and so informing the development of similar undertakings elsewhere. It was coordinated and chaired by the coordinator (RAND) with presentations by DISMEVAL partners from Austria (PMU), Denmark (UCPH), Germany (GUF; AQUA), France (UPVM), the Netherlands (UM) and Spain (ISCIII)
- A DISMEVAL workshop organised by the DISMEVAL Consortium at the 17th WONCA Europe 2011 Conference, Warsaw, Poland, 9 September 2011. The workshop aimed to discuss findings of a range of DISMEVAL work packages. It was coordinated and chaired by the Austrian DISMEVAL partner (PMU) with presentations by DISMEVAL partners from Austria (PMU), Germany (GUF), France (UPVM) and Spain (ISCIII)
2. Presentation of the project and/or individual work packages at national conferences including
- Annual Research Meeting of the German Society for Social Medicine (DGSMP) and the German Society for Medical Sociology (DGMS), Hamburg, Germany, 23-25 September 2009 (RAND)
- 2nd Conference on Chronic Care, Santiago de Compostela, Spain, 25-26 February 2010 (RAND)
- 2nd Forum Internacional Sobre Doentes Cronicos, Lisbon, Portugal, 1 May 2010 (RAND)
- Annual Danish Public Health Society Meeting 'Sundhedsdage 2010', Nyborg, Denmark, 29-30 June 2010 (UCPH)
- 44th Conference of the German Society of General Medicine DEGAM, Dresden, Germany, 23-25 September 2010 (PMU, GUF)
- 16th annual meeting of the Danish Forum for Health Services research, Copenhagen, Denmark, 25 November 2011 (UCPH)
- Austrian, German and Southern Tyrolian Society of General Medicine DEGAM, Salzburg, 21 September 2011 (PMU, GUF)
- GMDS/DGEpi 2011: Annual Meeting of the German Society for Medical Informatics, Biometrics and Epidemiology (GMDS) and the German Society for Epidemiology (DGEpi), Mainz, Germany, 26 September 2011 (AQUA)
- German Conference of Health Services Research, DKVF, Cologne, Germany, 20 October 2011 (AQUA)
- Chronic Disease and Health Management - 3rd edition 'Investing in the quality of health care', Bucharest, Romania, 7 October 2011 (RAND)
- Congress of the French society for Public Health, Lille, 2-4 November 2011 (UPVM)
3. Presentation of the project and/or individual work packages at international conferences including
- Presentation of the DISMEVAL project at the Academy Health Annual Research Meeting, Chicago, USA, 26-20 June 2009. This conference presented the first opportunity to present the aims and objectives of the DISMEVAL project to a wider audience of health services researchers and policy makers in North America, also acting as a catalyst to communicate European health services research to the research and policy community outside Europe (RAND)
- 2010 conference of 'The Future of Primary Health Care in Europe III', European Forum for Primary Care, Pisa , Italy, 8 August 2010 (poster: GUF)
- EURO-EPI 2010 'Epidemiology and Public Health in an Evolving Europe', Florence. Italy, 8 November 2011 (GUF)
- 4th European Public Health Conference 'Public Health and Welfare', Copenhagen, Denmark, 10-12 November 2011 (UCPH)
4. Invited (keynote) lectures at national and international meetings and conferences
- Hemispheric Meeting of the Social Protection and Health Network; Improving Chronic Disease Prevention and Management in Latin America and the Caribbean, Santiago de Chile, Chile, 30 September 2010. This meeting provided the unique opportunity to communicate and discuss findings emerging from DISMEVAL to senior policy makers in Latin America and the Caribbean, further acting as a means to communicate European research to the policy community outside Europe. (RAND)
- Belgian Presidency of the European Council 'Innovative approaches for chronic illness in public health and healthcare systems', Brussels, Belgium, 20 October 2010. This meeting, organised under the Belgian Presidency provide the opportunity to communicate and discuss findings emerging from the DISMEVAL to senior policy makers in the European Union (RAND)
- 3rd European Public Health Conference 'Integrated Public Health', Amsterdam, the Netherlands, 13 November 2010. This dissemination opportunity involved a plenary presentation at Europe's main conference for public health researchers, which brings together the public health research and policy community (RAND)
- International Seminar 'Becoming the Best: Building Sustainability High Performing Health Systems', Institute of Health Economics, Alberta, Canada, 15 April 2011 (RAND (P1))
- European Health Management Annual Conference 'Integration in Health and Health Care', Porto, Portugal, 23 June 2011. The annual EHMA conference brings together research, policy and management communities in the field of health management, so providing an opportunity. The dissemination opportunity involved a keynote lecture, so providing a means to communicate DISMEVAL findings to the health management community in particular (RAND)
- National Summit on Non communicable Diseases, Ministry of Health and Family Welfare and WHO Country Office for India, New Delhi, India, 23-24 August 2011. This national summit provided the unique opportunity to communicate and discuss findings from DISMEVAL to senior policy makers and researchers in India, further acting as a means to communicate European research to the research and policy community outside Europe. (RAND)
- Annual Conference Australian Disease Management Association, Canberra, Australia, 26 August 2011. This national conference provided the unique opportunity to communicate and discuss findings from DISMEVAL to senior policy makers and researchers in Australia, further acting as a means to communicate European research to the research and policy community outside Europe. (UM)
In addition to communicating emerging findings from the DISMEVAL project through presentations, peer-reviewed publications form an important route for dissemination. During the lifetime of the project, DISMEVAL partners have begun publishing in peer-reviewed journals.
These included:
1. Fullerton B, Nolte E, Erler A. Qualität der Versorgung chronisch Kranker in Deutschland. Z Evid Fortbild Qual Gesundhwes 2011;105:554-562. [Article in German]
2. Elissen AMJ., Duimel-Peeters IGM, Spreeuwenberg C, Vrijhoef HJM. Dismeval: Europees onderzoek naar 'best practices' op het gebied van disease management evaluatie. Tijdschrift voor Gezondheidswetenschappen 2011; 89:173-177. [Article in Dutch]
3. Flamm M, Panisch S, Winkler H, Sönnichsen AC. Impact of a randomized control group on perceived effectiveness of a Disease Management Programme for diabetes type 2. Eur J Public Health 2011 Oct 11 [Epub ahead of print]
4. Jacobsen R, Frølich A, Godtfredsen NS. Impact of exercise capacity on dyspnea and health-related quality of life in COPD patients. J Cardiopulm Rehabil Prev 2011 Dec 21. [Epub ahead of print]
5. Nolte E, Knai C, Hofmarcher M, Conklin A, Erler A, Elissen A, Flamm M, Fullerton B, Sönnichsen A, Vrijhoef HJ. Overcoming fragmentation in health care: chronic care in Austria, Germany and The Netherlands. Health Econ Policy Law 2012;7(1):125-46.
6. Fullerton B, Erler A, Pohlmann B, Gerlach FM. Predictors of dropout in the German disease management program for type 2 diabetes. BMC Health Serv Res 2012;12:8.
Furthermore, DISEMEVAL partners have led on and/or contributed to technical reports that are available to the public on the projects website (http://www.dismeval.eu/publications/ online for further details). These reports are:
1. Conklin A, Nolte E. Disease management evaluation. A comprehensive review of current state of the art. Santa Monica: RAND Corporation, 2011
2. Nolte E, Conklin A, Adams J, Brunn M, Cadier B, Chevreul K, Durand-Zaleski I, Elissen A, Erler A, Flamm M, Frølich A, Fullerton B, Jacobsen R, Knai C, Krohn R, Pöhlmann B, Saz Parkinson Z, Sarria Santamera A, Sönnichsen A, Vrijhoef H. Evaluating chronic disease management. Recommendations for funders and users. RAND Europe/DISMEVAL Consortium, 2012.
3. Nolte E, Hinrichs S, eds. DISMEVAL. Developing and validating disease management evaluation methods for European healthcare systems. Final report. RAND Europe/DISMEVAL Consortium, 2012.
Dissemination is expected to be ongoing beyond the formal project period with a series of consortium publications underway/close to submission including a volume to be published as a monograph in summer 2012, reporting on 'Chronic disease management in Europe - An overview of 13 countries'.
List of Websites:
http://www.dismeval.eu