Periodic Reporting for period 4 - EPP (Econometrics for Public Policy: Sampling, Estimation, Decision, and Applications)
Reporting period: 2021-08-01 to 2022-01-31
The importance of evidence-based policy making is widely recognized amongst policy makers worldwide. A vast amount of empirical work in economics aims to achieve this goal, but the current literature of econometrics provides rather limited recommendations for estimating a welfare maximizing policy. The main goal of this project is to develop statistically optimal and practically attractive ways of learning welfare maximising policies from the data, which can inform the design of public policies and decision making by policy makers.
The project made contributions to the following three aspects of evidence-based policy design:
(1) Estimation of causal effects and optimal policies: what estimation procedure should be used to estimate optimal policies?
We proposed empirical welfare maximization (EWM) methods for estimating an optimal targeting policy that prescribes who should get what policy. We clarified that the EWM-based optimal targeting policy shows its statistical optimality in terms of welfare performances of the estimated policy. The framework allowed a large class of social welfare criterion, and indicated how to make use of machine learning algorithms for the design of treatment allocation policies. We also developed inference methods for the welfare attained by an estimated policy. The main contributions appear in the following papers:
- "Who Should Be Treated? Empirical Welfare Maximization Methods for Treatment Choice," with Aleksey Tetenov, (2018), Econometrica
- "Equality-minded Treatment Choice," with Aleksey Tetenov, (2021), Journal of Business and Economic Statistics
- "Inference on Winners," with Isaiah Andrews and Adam McCloskey, (2024). Quarterly Journal of Economics
(2) Decision making: what is the optimal decision-making procedure given uncertainty about the welfare impact of the policies?
We developed a novel framework for binary policy choice based on nonlinear welfare regret criterion, and showed that Bayes and minimax optimal policy choices are randomized decisions. We also studied treatment choice in a meta-analysis context, and clarifies a set of conditions under which computation of the minimax regret decision can be simplified.
For general cases where the data and assumptions cannot pin down a unique point estimate for causal impact of policy (set-identified causal effect), we studied inference for it from the robust Bayes viewpoint. This approach can mitigate a common concern of prior dependence in Bayesian inference and at the same time, we showed it achieves desired frequentist performances asymptotically. The main contributions appear in the following papers.
- "Evidence Aggregation for Treatment Choice," with Takuya Ishihara, (2021), arXiv.
- "Treatment Choice with Nonlinear Regret," with Sokbae Lee and Chen Qiu (2023), arXiv
- "Robust Bayesian Inference for Set-Identified Models," with Raffaella Giacomini, (2021), Econometrica
(3) Applications to reality: Can the developed methods apply to the real-world?
To apply the methods developed in this project, we focused on targeted rebate policy in electricity market.
We ran a field experiment in Japan to learn the heterogeneous household demand response of electricity consumption to electricity pricing. Using this experimental data, we applied the methods developed in (1) to estimate optimal targeting of rebate program of electricity conservation, and showed that the targeted assignments of the rebates can significantly reduce the dead-weight loss in electricity market. The completed manuscript is "Choosing Who Chooses: Selection-Driven Targeting in Energy Rebate Programs" by Ida, Ishihara, Ito, Kitagawa, Sakaguchi, and Sasaki (2023, NBER working paper), which is currently revise and resubmit at Econometrica.
The project made contributions to the following three aspects of evidence-based policy design:
(1) Estimation of causal effects and optimal policies: what estimation procedure should be used to estimate optimal policies?
We proposed empirical welfare maximization (EWM) methods for estimating an optimal targeting policy that prescribes who should get what policy. We clarified that the EWM-based optimal targeting policy shows its statistical optimality in terms of welfare performances of the estimated policy. The framework allowed a large class of social welfare criterion, and indicated how to make use of machine learning algorithms for the design of treatment allocation policies. We also developed inference methods for the welfare attained by an estimated policy. The main contributions appear in the following papers:
- "Who Should Be Treated? Empirical Welfare Maximization Methods for Treatment Choice," with Aleksey Tetenov, (2018), Econometrica
- "Equality-minded Treatment Choice," with Aleksey Tetenov, (2021), Journal of Business and Economic Statistics
- "Inference on Winners," with Isaiah Andrews and Adam McCloskey, (2024). Quarterly Journal of Economics
(2) Decision making: what is the optimal decision-making procedure given uncertainty about the welfare impact of the policies?
We developed a novel framework for binary policy choice based on nonlinear welfare regret criterion, and showed that Bayes and minimax optimal policy choices are randomized decisions. We also studied treatment choice in a meta-analysis context, and clarifies a set of conditions under which computation of the minimax regret decision can be simplified.
For general cases where the data and assumptions cannot pin down a unique point estimate for causal impact of policy (set-identified causal effect), we studied inference for it from the robust Bayes viewpoint. This approach can mitigate a common concern of prior dependence in Bayesian inference and at the same time, we showed it achieves desired frequentist performances asymptotically. The main contributions appear in the following papers.
- "Evidence Aggregation for Treatment Choice," with Takuya Ishihara, (2021), arXiv.
- "Treatment Choice with Nonlinear Regret," with Sokbae Lee and Chen Qiu (2023), arXiv
- "Robust Bayesian Inference for Set-Identified Models," with Raffaella Giacomini, (2021), Econometrica
(3) Applications to reality: Can the developed methods apply to the real-world?
To apply the methods developed in this project, we focused on targeted rebate policy in electricity market.
We ran a field experiment in Japan to learn the heterogeneous household demand response of electricity consumption to electricity pricing. Using this experimental data, we applied the methods developed in (1) to estimate optimal targeting of rebate program of electricity conservation, and showed that the targeted assignments of the rebates can significantly reduce the dead-weight loss in electricity market. The completed manuscript is "Choosing Who Chooses: Selection-Driven Targeting in Energy Rebate Programs" by Ida, Ishihara, Ito, Kitagawa, Sakaguchi, and Sasaki (2023, NBER working paper), which is currently revise and resubmit at Econometrica.
This project has made several important contributions to methodological foundations, analytical and computational developments, and empirical practice of evidence-based policy making. We proposed novel methods of informing evidence-based treatment assignment policies based on experimental and observational data, which we named as Empirical Welfare Maximization (EWM) methods. We clarified analytical and computational properties of the EWM methods and implemented it to learn allocation of job training programs and assignment of rebate programs for electricity conservation.
To make the EWM methods appealing for the real policy practice, the project also made several extensions of the methods. They include incorporation of equality-minded welfare criterion, modified adaptation of machine learning algorithms, unification with adaptive sampling design, statistical inference for optimized welfare, policy decision under ambiguity on welfare criterion, spillover and equilibrium effects of policy assignments, and policy maker's preference for regret aversion. In terms of academic achievements, the results of the project has created a field and trend of econometrics research, now labeled as "policy learning." The extensions also offer practitioners a package of tools for evidence-based policy making.
Since the beginning of the funding period, we have been presenting the research agenda and results in numerous seminars and conferences. I had opportunities to deliver invited lecture on this topic in Kyoto University and Humbolt University in Berlin. I gave keynote talks on this topic in a couple of Econometric Society Meetings. I incorporated the progress of the project into teaching materials for PhD courses in UCL and Brown University.
To make the EWM methods appealing for the real policy practice, the project also made several extensions of the methods. They include incorporation of equality-minded welfare criterion, modified adaptation of machine learning algorithms, unification with adaptive sampling design, statistical inference for optimized welfare, policy decision under ambiguity on welfare criterion, spillover and equilibrium effects of policy assignments, and policy maker's preference for regret aversion. In terms of academic achievements, the results of the project has created a field and trend of econometrics research, now labeled as "policy learning." The extensions also offer practitioners a package of tools for evidence-based policy making.
Since the beginning of the funding period, we have been presenting the research agenda and results in numerous seminars and conferences. I had opportunities to deliver invited lecture on this topic in Kyoto University and Humbolt University in Berlin. I gave keynote talks on this topic in a couple of Econometric Society Meetings. I incorporated the progress of the project into teaching materials for PhD courses in UCL and Brown University.
The results of this project laid out theoretical foundations of econometrics for evidence-based policy making, and paved paths though which econometrics can tightly link to welfare economics, machine learning, and economic decision theory. Also the proposed perspective of shifting the focus from statistical loss criterion to economic welfare criterion opened up various novel research questions that make econometrics closer to economics.
The novelty and importance of the contributions have been widely recognized, as indicated by top journal publications out of this project including Econometrica, Quarterly Journal of Economics, Journal of Econometrics, Journal of Business Economics and Statistics, Quantitative Economics, and Journal of Applied Econometrics.
The novelty and importance of the contributions have been widely recognized, as indicated by top journal publications out of this project including Econometrica, Quarterly Journal of Economics, Journal of Econometrics, Journal of Business Economics and Statistics, Quantitative Economics, and Journal of Applied Econometrics.