In this project, we push the frontier of knowledge required for effective and efficient policy interventions to battle antibiotic resistance.
We begin by providing causal evidence of the role of physicians in general practice in driving inefficiencies in antibiotic prescribing. We show that variation in antibiotic prescribing intensity is largely driven by differences in physician practice styles. We document that variation in prescribing is not related to health outcomes, suggesting that the documented variation does not reflect varying ability to avoid adverse health outcomes due to potential undertreatment. Building on this result, in the project we analyze the potential mechanisms leading to differences in practice styles, in particular differences in diagnostic information and preferences towards the curbing of antibiotic resistance.
At the core of the project, we proceed establish the relationship between bacterial infections and information encoded in historical administrative patient-level data. This relationship is crucial for understanding the effectiveness of policies aimed at curbing antibiotic resistance as well as the value of information for medical decision making. The basis for our analysis are microbiological laboratory data linked with administrative data from Denmark. We first implement a machine learning algorithm to predict the presence of bacteria in test samples for patients with suspected urinary tract infection in general practice. Based on this, we measure the potential effectiveness of a machine learning-based policy aimed at reducing antibiotic prescribing. For suspected urinary tract infections, we find that overall prescribing could be reduced by approximately 10% and overprescribing, defined as an antibiotic prescription to a person without a bacterial infection, by 25%.
Building on black box machine learning predictions, we provide insights into the challenges of evaluating such policies prospectively. First, we show that a common challenge for out-of-sample predictions, distribution shift, carries over to out-of-sample policy evaluation. We document that ignoring variation in distribution of consulting patients over time can lead to misleading policy conclusions. We propose a simple rule, updating policy parameters over time, that provides robust out-of-sample policy improvements. Second, analyze the value of increasing the scope of data used for prediction. We compare the common practice in evaluating the potential of machine learning, focussing on prediction quality, to basing evaluations on the policy objective function. We document that the added value of data differs for prediction and policy outcomes. We emphasize that machine learning predictions contribute the largest value for decision outcomes, when data likely unavailable or too complex for analysis in clinical practice are used.
Finally, we propose a model that allows the measurement of heterogeneous physician diagnostic skill and incentives, when patient groups vary. Based on this model, we quantify the heterogeneity in (unobservable) clinical diagnostic information physicians use as well as in the weight they place on the antibiotic resistance externality. We evaluate the counterfactual provision of machine learning predictions to physicians in general practice and find that combining these predictions with physicians' clinical diagnostic information does not lead to a reduction in overall prescribing given physicians' estimated weight on the antibiotic resistance externality.
This result shows that improvements suggested by a simple prediction-based decision rule are not achieved by the superior quality of machine learning predictions but are due to the imposition of the policy maker's objective function. We find that the same reductions in prescribing as with the prediction-based decision rule can be achieved by further incentivizing physicians to place a higher weight on the externality.
Trials to take the machine learning-based predictions into clinical practice are in preparation.
