Periodic Reporting for period 3 - ABRSEIST (Antibiotic Resistance: Socio-Economic Determinants and the Role of Information and Salience in Treatment Choice)
Reporting period: 2022-01-01 to 2023-06-30
Antibiotics have contributed to a tremendous increase in human well-being, saving many millions of lives. However, antibiotics become obsolete the more they are used as selection pressure promotes the development of resistant bacteria. The World Health Organization has proclaimed antibiotic resistance as a major global threat to public health. Today, 700,000 deaths per year are due to untreatable infections. To win the battle against antibiotic resistance, new policies affecting the supply and demand of existing and new drugs must be designed. In this project, we pursue new research to identify and evaluate feasible and effective demand-side policy interventions targeting the relevant decision makers: physicians and patients. ABRSEIST makes use of a broad econometric toolset to identify mechanisms linking antibiotic resistance and consumption exploiting a unique combination of physician-patient-level antibiotic resistance, treatment, and socio-economic data. Further, it aims to shed light on general practitioners’ acquisition and use of information under uncertainty about resistance in prescription choice, allowing counterfactual analysis of information-improving policies such as the provision of machine learning predictions in clinical practice and mandatory diagnostic testing. Using machine learning methods, theory-driven structural econometric analysis, and randomization in the field we work to provide rigorous evidence on effective intervention designs. This research will improve our understanding of how prescribing, resistance, and the effect of antibiotic use on resistance, are distributed in the general population which has important implications for the design of targeted interventions.
In this project, we push the frontier of knowledge required for effective and efficient policy interventions to battle antibiotic resistance.
We begin by providing causal evidence of the role of physicians in general practice in driving inefficiencies in antibiotic prescribing. We show that variation in antibiotic prescribing intensity is largely driven by differences in physician practice styles. We document that variation in prescribing is not related to health outcomes, suggesting that the documented variation does not reflect varying ability to avoid adverse health outcomes due to potential undertreatment. Building on this result, in the project we analyze the potential mechanisms leading to differences in practice styles, in particular differences in diagnostic information and preferences towards the curbing of antibiotic resistance.
At the core of the project, we proceed establish the relationship between bacterial infections and information encoded in historical administrative patient-level data. This relationship is crucial for understanding the effectiveness of policies aimed at curbing antibiotic resistance as well as the value of information for medical decision making. The basis for our analysis are microbiological laboratory data linked with administrative data from Denmark. We first implement a machine learning algorithm to predict the presence of bacteria in test samples for patients with suspected urinary tract infection in general practice. Based on this, we measure the potential effectiveness of a machine learning-based policy aimed at reducing antibiotic prescribing. For suspected urinary tract infections, we find that overall prescribing could be reduced by approximately 10% and overprescribing, defined as an antibiotic prescription to a person without a bacterial infection, by 25%.
Building on black box machine learning predictions, we provide insights into the challenges of evaluating such policies prospectively. First, we show that a common challenge for out-of-sample predictions, distribution shift, carries over to out-of-sample policy evaluation. We document that ignoring variation in distribution of consulting patients over time can lead to misleading policy conclusions. We propose a simple rule, updating policy parameters over time, that provides robust out-of-sample policy improvements. Second, analyze the value of increasing the scope of data used for prediction. We compare the common practice in evaluating the potential of machine learning, focussing on prediction quality, to basing evaluations on the policy objective function. We document that the added value of data differs for prediction and policy outcomes. We emphasize that machine learning predictions contribute the largest value for decision outcomes, when data likely unavailable or too complex for analysis in clinical practice are used.
Finally, we propose a model that allows the measurement of heterogeneous physician diagnostic skill and incentives, when patient groups vary. Based on this model, we quantify the heterogeneity in (unobservable) clinical diagnostic information physicians use as well as in the weight they place on the antibiotic resistance externality. We evaluate the counterfactual provision of machine learning predictions to physicians in general practice and find that combining these predictions with physicians' clinical diagnostic information does not lead to a reduction in overall prescribing given physicians' estimated weight on the antibiotic resistance externality.
This result shows that improvements suggested by a simple prediction-based decision rule are not achieved by the superior quality of machine learning predictions but are due to the imposition of the policy maker's objective function. We find that the same reductions in prescribing as with the prediction-based decision rule can be achieved by further incentivizing physicians to place a higher weight on the externality.
Trials to take the machine learning-based predictions into clinical practice are in preparation.
We begin by providing causal evidence of the role of physicians in general practice in driving inefficiencies in antibiotic prescribing. We show that variation in antibiotic prescribing intensity is largely driven by differences in physician practice styles. We document that variation in prescribing is not related to health outcomes, suggesting that the documented variation does not reflect varying ability to avoid adverse health outcomes due to potential undertreatment. Building on this result, in the project we analyze the potential mechanisms leading to differences in practice styles, in particular differences in diagnostic information and preferences towards the curbing of antibiotic resistance.
At the core of the project, we proceed establish the relationship between bacterial infections and information encoded in historical administrative patient-level data. This relationship is crucial for understanding the effectiveness of policies aimed at curbing antibiotic resistance as well as the value of information for medical decision making. The basis for our analysis are microbiological laboratory data linked with administrative data from Denmark. We first implement a machine learning algorithm to predict the presence of bacteria in test samples for patients with suspected urinary tract infection in general practice. Based on this, we measure the potential effectiveness of a machine learning-based policy aimed at reducing antibiotic prescribing. For suspected urinary tract infections, we find that overall prescribing could be reduced by approximately 10% and overprescribing, defined as an antibiotic prescription to a person without a bacterial infection, by 25%.
Building on black box machine learning predictions, we provide insights into the challenges of evaluating such policies prospectively. First, we show that a common challenge for out-of-sample predictions, distribution shift, carries over to out-of-sample policy evaluation. We document that ignoring variation in distribution of consulting patients over time can lead to misleading policy conclusions. We propose a simple rule, updating policy parameters over time, that provides robust out-of-sample policy improvements. Second, analyze the value of increasing the scope of data used for prediction. We compare the common practice in evaluating the potential of machine learning, focussing on prediction quality, to basing evaluations on the policy objective function. We document that the added value of data differs for prediction and policy outcomes. We emphasize that machine learning predictions contribute the largest value for decision outcomes, when data likely unavailable or too complex for analysis in clinical practice are used.
Finally, we propose a model that allows the measurement of heterogeneous physician diagnostic skill and incentives, when patient groups vary. Based on this model, we quantify the heterogeneity in (unobservable) clinical diagnostic information physicians use as well as in the weight they place on the antibiotic resistance externality. We evaluate the counterfactual provision of machine learning predictions to physicians in general practice and find that combining these predictions with physicians' clinical diagnostic information does not lead to a reduction in overall prescribing given physicians' estimated weight on the antibiotic resistance externality.
This result shows that improvements suggested by a simple prediction-based decision rule are not achieved by the superior quality of machine learning predictions but are due to the imposition of the policy maker's objective function. We find that the same reductions in prescribing as with the prediction-based decision rule can be achieved by further incentivizing physicians to place a higher weight on the externality.
Trials to take the machine learning-based predictions into clinical practice are in preparation.
In this project, we have pushed the frontier beyond the state of the art in several ways, filling several gaps in the scientific evidence on antibiotic prescribing and resistance as well as providing pivotal policy-relevant insights. First, we provide rigorous causal evidence for the importance of heterogeneous practice styles in determining antibiotic consumption intensities. Second, we show the value of administrative data for developing policies that improve the efficiency of antibiotic prescribing in general practice. We highlight pitfalls in the evaluation of counterfactual policies and provide ways to overcome these. Third, we show how physician skill, complementary to machine learning predictions, can be identified when patient groups, physician skill, and physician preferences vary across clinics in general practice.
The main pillars until the end of the project will be to provide causal evidence on the importance of human antibiotic consumption for antibiotic resistance, to investigate further behavioral drivers of physicians' antibiotic prescribing styles, and to evaluate machine predictions in the field, in general practice.
The main pillars until the end of the project will be to provide causal evidence on the importance of human antibiotic consumption for antibiotic resistance, to investigate further behavioral drivers of physicians' antibiotic prescribing styles, and to evaluate machine predictions in the field, in general practice.