Periodic Reporting for period 3 - EPP (Econometrics for Public Policy: Sampling, Estimation, Decision, and Applications) Reporting period: 2020-02-01 to 2021-07-31 Summary of the context and overall objectives of the project The importance of evidence-based policy making is widely recognized amongst policy makers worldwide and one of the ultimate goals of economics is to inform a policy in a manner that improves welfare. A vast amount of empirical work in economics aims to achieve this goal, but the current literature of econometrics provides rather limited recommendations for estimating a welfare maximizing policy. This project concerns statistically optimal and practically attractive ways of learning welfare maximising policies from the data. It does so by developing novel econometric frameworks, sample design methodologies and estimation approaches, all of which can be applied to a wide variety of real-world policy design problems. The main goal of the project is to develop econometric methods that can aid in the design of public policies and decision making by policy makers. It specifically focuses on the following four aspects of evidence-based policy design:(1) Sampling design: what are better ways to collect data to learn an optimal policy? (2) Estimation of causal effects and optimal policies: what estimation procedure should be used to estimate optimal policies? (3) Decision making: what is the optimal decision-making procedure given uncertainty about the welfare impact of the policies?(4) Applications to reality: Can the procedures developed in (1) - (3) be applied to the real-world? Building on theoretical and computational tools from econometrics, decision theory, and machine learning, this project aims to provide a framework and practical procedures for evidence-based policy design. A large part of the project focuses on the design of treatment assignment policies over heterogeneous individuals, so called targeting policy, personalized medicine, individualized treatment, etc. in the literature. Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far This section reviews the specific projects that I have been running or completed as of April 2020. I classify these projects according to the aspects (1) - (4) described above and list them in order below. This numbering will also be used when I refer to particular in the later sections of the report. (1) Sampling design: what are better ways to collect data to learn an optimal policy? (a) Welfare optimal design of field experiments (with Aleksey Tetenov, RA: Jeff Rowley), (b) Treatment assignment with reinforcement learning (with Thomas Carr and Kohei Kawaguchi),Project (a) focuses on methods of determining the sampling design in field experiments such that it leads to an estimate of a policy that attains a high level of social welfare. Project (b) considers dynamic settings where one can sample observations, learn, and implement policies simultaneously over multiple time periods.(2) Estimation of causal effects and optimal policies: what estimation procedure should be used to estimate optimal policies? (a) Development of empirical welfare maximization methods (with Aleksey Tetenov), published in Econometrica in 2018. (b) Equality-minded treatment choice (with Aleksey Tetenov), published in Journal of Business Economics and Statistics in 2019. (c) Validity of empirical Monte Carlo studies for causal inference (with Arun Advani and Tymon Slozynski), published in Journal of Applied Econometrics in 2019 (d) Inference for social welfare attained by estimated policy (with Isaiah Andrews and Adam McCloskey), Submitted (e) Constrained machine learning for treatment choice using surrogate loss (with Shosei Sakaguchi and Aleksey Tetenov) (f) Stochastic treatment choice (with Hugo Lopez-Lopez). (g) Boosting algorithms for treatment choice (with Yu-Lin Lin) Completed projects (a) and (b) concern the estimation of an optimal targeting policy under a variety of social welfare functions. These papers are some of the major achievements of this project so far. Completed project (c) examines whether Monte Carlo simulations can be used to assess the performance of empirical methods. Completed but not yet published project (d) proposes a method for inferring the social welfare of an estimated policy. Building on (a) and (b), ongoing projects (e), (f), and (g) examine how classification algorithms from machine learning can be applied to the estimation of a targeting policy. (3) Decision making: what is the optimal decision-making procedure given uncertainty about the welfare impact of the policies? (a) Robust Bayesian inference for set-identified models (with Raffaella Giacomini), R&R, Econometrica (b) Estimation under ambiguity (with Raffaella Giacomini and Harald Uhlig), R&R, Quantitative Economics (c) Model averaging set- and point-identified models (with Raffaella Giacomini and Alessio Volpicella), R&R, Quantitative Economics (d) Posterior distribution of nondifferentiable functions (with Jose Montiel-Olea, Jonathan Payne, and Amilcar Velez), forthcoming in Journal of Econometrics (e) Evidence aggregation and meta-analysis for treatment choice (with Takuya Ishihara and Masayuki Sawada)Completed projects (a) - (d), among other things, provide frameworks and analytical results for decision-making when the available data only set-identifies the welfare criterion. Ongoing project (e) analyses the decision of whether to adopt a policy or not in a meta-analysis setting where the decision maker has no data for the population of interest but has access to a set of existing studies performed in different populations. (4) Applications to reality: Can the procedures developed in (1) - (3) be applied to the real-world? (a) Targeted rebate policy in electricity market (with Takanori Ida, Koichiro Ito, and Daido Kido) (b) Personalized nudge to charity donors (with Syusaku Sasaki, Takanori Ida, Takunori Ishihara, and Daido Kido) (c) Targeted tax auditing (with Arun Advani, RA: Guanyi Wang) With the group of researchers in Kyoto University, I initiated empirical projects (a) and (b) that apply the methods developed in the projects (2-a) to real-world policy issues. In (a), we are running a field experiment in Japan to learn the heterogeneous household price elasticity of electricity consumption. The goal of this project is to design a personalized rebate policy that pays back cash to households depending on how much they reduce their electricity consumption relative to their individual baseline level. The goal of this policy is to control electricity demand and reduce CO2 emissions. The methods proposed in Project (2-a) are used to estimate a personalized rebate policy that minimizes the welfare loss of the price intervention. Project (b) concerns an application similar to the setting of (1-b). We have conducted online experiments to examine the heterogeneity of individual donation responses to nudges such as seed-money matching and social comparison. Using this data, we plan to assess whether reinforcement learning algorithms such as bandit algorithms are effective at dynamically learning personalized nudge strategies and increasing charity donations. Project (c) applies the empirical welfare maximization methods developed in (2-a) to design tax audit policy. We obtained access to tax audit data held by HMRC (the UK tax authority), and plan to estimate an efficient targeting policy subject to capacity and cost constraints. Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far) The completed projects indicated in the previous section have either been published or reached the revise & resubmit stage in top journals, such as Econometrica, Journal of Econometrics, Journal of Business Economics and Statistics, Quantitative Economics, and Journal of Applied Econometrics.Projects (2-a) and (2-b) made methodological contributions to the literature of statistical treatment choice by proposing estimation procedures for treatment assignment policies and showing that they are welfare-optimal. Project (2-c) showed that using empirical Monte Carlo methods is not a valid way of finding the best causal effect estimator. This finding cautions against the use of empirical Monte Carlo procedures in the context of estimating optimal treatment assignment policies. The completed projects in (3) propose a framework and a set of novel procedures for robust Bayes inference in set-identified models. These results can be useful for (but are not limited to) the analysis of open problems in the literature concerning policy decisions in the presence of a set-identified welfare criterion. By the end of the final period of this project, I aim to complete the ongoing projects listed above, and to at least have the draft papers ready for submission to academic journals. All these projects aim to provide novel perspectives and important contributions to the literature. I expect that the majority of them will be published in either a top general journal in economics or a top field journal in econometrics, as has been the case for the projects completed so far.
