Periodic Reporting for period 1 - REALLYCREDIBLE (Completing the revolution : Enhancing the reality, the principles, and the impact of economics' credibility revolution)
Reporting period: 2022-11-01 to 2025-04-30
Social scientists routinely evaluate the effect of economic policies. For instance, what is the effect of raising the minimum wage on employment? Angrist and Pischke (2010) argue that quantitative social sciences, and in particular applied economics, have recently experienced a “credibility revolution'”: by switching to transparent methods to evaluate policies, social scientists have increased the credibility of their findings, and the impact of their work. However, the credibility revolution is not complete. Two-way fixed effects regression (TWFE) is a policy-evaluation method used in as many as 26% of the most-highly cited papers recently published in the American Economic Review. Social scientists have long believed that this method is similar to a difference-in-differences (DID) method. DID yields an unbiased estimator of the policy’s effect under a parallel trends assumption: without the treatment, treated and control units would have experienced the same outcome evolution. Parallel trends is an appealing assumption, because it is partly testable: one can check whether prior to treatment, treated and control units were experiencing parallel outcome evolutions. However, recent papers have shown that TWFE regressions also rely on the assumption that the treatment effect is constant, both across units and over time. This constant-effect assumption is implausible in most applications. For instance, the effect of the minimum wage on employment could be more negative in locations with a less qualified workforce, and that effect could also change over time. This constant-effect assumption is also much harder to test than parallel trends. Therefore, the project proposes a series of so-called “generalized DID estimators” (GDID), that only rely on parallel trends assumptions and do not rely on the constant effect assumption, and that can be used in virtually all the applications where two-way fixed effects regressions have been used. The project also proposes software implementation of those new estimators, in Stata and R, the two statistical software most commonly used by quantitative social scientists. Thus, it hopes to provide easily accessible tools that quantitative social scientists can use to produce more credible policy evaluations.
So far, the project has delivered one published and one forthcoming paper:
• Two-way Fixed Effects and Differences-in-Differences Estimators with Several Treatments, Journal of Econometrics, 2023, Paper 1. This paper highlights issues with TWFE regressions when one wants to study the effect of several rather than one policy, and proposes alternative GDID tailored to that case.
• “Difference-in-Differences Estimators of Intertemporal Treatment Effects”, Forthcoming, Review of Economics and Statistics, Paper 2. This paper proposes GDID when units’ outcome is affected by the current economic policy they face, but also by the economic policies they faced in the past. Using the proposed estimators, the paper finds long-lasting effects of the financial deregulations conducted in the US in the 1980s on the volume of credit lent by banks, and on houses prices. Using TWFE regressions, previous literature had found short-lived effects.
The project has also delivered one paper published in the proceedings of an important conference:
• “Difference-in-Differences Estimators with Continuous Treatments and no Stayers”, American Economic Association Papers & Proceedings, 2024, Paper 3. This paper highlights difficulties with constructing GDID estimators with continuous treatments and no stayers (units whose treatment does not change over time). It sill proposes a parametric alternative to the classical TWFE estimator, and uses it to reestimate the effect of rising temperatures on agricultural yields in the US.
The project has also delivered five working papers:
• “More Robust Estimators for Instrumental-Variable Panel Designs, With An Application to the Effect of Chinese Imports on US Employment” (Paper 4)
• “Difference-in-Differences for Continuous Treatments and Instruments with Stayers” (Paper 5).
• “Two-way Fixed Effects and Difference-in-Difference Estimators in Heterogeneous Adoption Designs without Stayers” (Paper 6).
• “Trading-off Bias and Variance in Stratified Randomized Controlled Trials, Under a Boundedness Condition on the Magnitude of the Treatment Effect” (Paper 7).
• “Estimating treatment-effect heterogeneity across sites, in multi-site randomized experiments with few units per site” (Paper 8).
• Two-way Fixed Effects and Differences-in-Differences Estimators with Several Treatments, Journal of Econometrics, 2023, Paper 1. This paper highlights issues with TWFE regressions when one wants to study the effect of several rather than one policy, and proposes alternative GDID tailored to that case.
• “Difference-in-Differences Estimators of Intertemporal Treatment Effects”, Forthcoming, Review of Economics and Statistics, Paper 2. This paper proposes GDID when units’ outcome is affected by the current economic policy they face, but also by the economic policies they faced in the past. Using the proposed estimators, the paper finds long-lasting effects of the financial deregulations conducted in the US in the 1980s on the volume of credit lent by banks, and on houses prices. Using TWFE regressions, previous literature had found short-lived effects.
The project has also delivered one paper published in the proceedings of an important conference:
• “Difference-in-Differences Estimators with Continuous Treatments and no Stayers”, American Economic Association Papers & Proceedings, 2024, Paper 3. This paper highlights difficulties with constructing GDID estimators with continuous treatments and no stayers (units whose treatment does not change over time). It sill proposes a parametric alternative to the classical TWFE estimator, and uses it to reestimate the effect of rising temperatures on agricultural yields in the US.
The project has also delivered five working papers:
• “More Robust Estimators for Instrumental-Variable Panel Designs, With An Application to the Effect of Chinese Imports on US Employment” (Paper 4)
• “Difference-in-Differences for Continuous Treatments and Instruments with Stayers” (Paper 5).
• “Two-way Fixed Effects and Difference-in-Difference Estimators in Heterogeneous Adoption Designs without Stayers” (Paper 6).
• “Trading-off Bias and Variance in Stratified Randomized Controlled Trials, Under a Boundedness Condition on the Magnitude of the Treatment Effect” (Paper 7).
• “Estimating treatment-effect heterogeneity across sites, in multi-site randomized experiments with few units per site” (Paper 8).
The work generated by the project is receiving a lot of interest, as shown by the metrics below. Together, the five aforementioned papers already have 938 Google Scholar citations. The tweet announcing the release of Paper 4 was retweeted 284 times, liked 836 times, and seen by 713 000 users. In July of 2023, we released did_multiplegt_dyn, a Stata implementation of the estimators proposed in Paper 2. Last month, this package was downloaded 3400 times, thus making it the 70th most downloaded Stata package on the SSC repository. My ERC team has now produced 9 Stata and R packages. When outputting results to the user, all the software produced by the project mention that their development was funded by the ERC grant.
To ensure further uptake and success, it is key to update and improve those packages, especially the R ones, which could be coded more efficiently to ensure that the computing time is reduced. I am currently supervising a team of three research assistants for that purpose. Once they are ready, effectively advertising those tools on social media (bluesky) is also key for success.
To ensure further uptake and success, it is key to update and improve those packages, especially the R ones, which could be coded more efficiently to ensure that the computing time is reduced. I am currently supervising a team of three research assistants for that purpose. Once they are ready, effectively advertising those tools on social media (bluesky) is also key for success.