Periodic Reporting for period 1 - Competing Forecasts (Comparing the Predictive Ability of Forecasting Models)
Berichtszeitraum: 2018-09-01 bis 2020-08-31
This project studies the problem of identification failure in a variety of econometric models. The main aim is to analyze the impact of this issue on estimation, inference and forecast comparison, using several models. The research shows that whenever the model is not strongly identified, the finite sample distributions of estimators and test statistics are affected, in the sense that these can be nonstandard. This might result then in misleading estimation and inference, which could eventually affect policy decisions when using these models at central banks, or other policy institutions. In order to overcome these issues, the projects proposes robust methods for inference.
From a theoretical point of view, identification of econometric models is crucial for valid estimation, inference and testing. From a policy perspective, as the policy implications of observational equivalent parameter points can be distinct, the reliability of policy recommendations rests upon the primitive assumptions of identifiability. This could potentially lead to important problems, if the researcher does not acknowledge the issue, but assumes a standard normal distribution. Decisions about the significance of parameters, and confidence intervals could be misleading. Thus, this research promotes the importance of developing tools and methods that are robust to this issue. Results emerging from this project are of interest to a large international academic community interested in predictive ability evaluation, central banks and other governmental organizations that could take-up the new knowledge for policy decisions, as well as to forecasting institutions and businesses that produce predictions which could improve their evaluation methodologies. These governmental and industry institutions are direct potential users of the project results.
The following scientific overall objectives have been achieved:
- The project shows that whenever the model is not strongly identified, the finite sample properties of estimators are affected. The distribution of the estimators for different parameters will have nonstandard/unexpected shapes: bimodal, uniform distributions appear quite often.
- Similarly, the distribution of tests such as the t-test or the likelihood ratio tests are also affected. The identification issues transfer from estimators to tests. Thus, decisions about the significance of parameters and confidence intervals could be misleading.
- The problem can also appear when comparing the predictive ability of different models and thus the finite sample and asymptotic properties of predictive ability tests are studied.
- In order to circumvent the problem three robust critical values are proposed. These rely on the actual finite sample distribution of tests and are shown to perform well in simulations and applications.
- Two empirical application showcase the prevalence of this issue in financial and macroeconomic data and illustrate the use of the proposed critical values.
From a theoretical point of view, identification of econometric models is crucial for valid estimation, inference and testing. From a policy perspective, as the policy implications of observational equivalent parameter points can be distinct, the reliability of policy recommendations rests upon the primitive assumptions of identifiability. This could potentially lead to important problems, if the researcher does not acknowledge the issue, but assumes a standard normal distribution. Decisions about the significance of parameters, and confidence intervals could be misleading. Thus, this research promotes the importance of developing tools and methods that are robust to this issue. Results emerging from this project are of interest to a large international academic community interested in predictive ability evaluation, central banks and other governmental organizations that could take-up the new knowledge for policy decisions, as well as to forecasting institutions and businesses that produce predictions which could improve their evaluation methodologies. These governmental and industry institutions are direct potential users of the project results.
The following scientific overall objectives have been achieved:
- The project shows that whenever the model is not strongly identified, the finite sample properties of estimators are affected. The distribution of the estimators for different parameters will have nonstandard/unexpected shapes: bimodal, uniform distributions appear quite often.
- Similarly, the distribution of tests such as the t-test or the likelihood ratio tests are also affected. The identification issues transfer from estimators to tests. Thus, decisions about the significance of parameters and confidence intervals could be misleading.
- The problem can also appear when comparing the predictive ability of different models and thus the finite sample and asymptotic properties of predictive ability tests are studied.
- In order to circumvent the problem three robust critical values are proposed. These rely on the actual finite sample distribution of tests and are shown to perform well in simulations and applications.
- Two empirical application showcase the prevalence of this issue in financial and macroeconomic data and illustrate the use of the proposed critical values.
The performed work can be summarized in the following work packages. This constitutes an overview of the main results.
Work Package 1: The most recent related literature has been surveyed thoroughly and linked to the project. Similarities and differences with respect to other papers have been summarized and clarified.
Work Package 2: The asymptotic distribution of predictive ability tests under loss of strong identification has been derived.
Work Package 3: As a solution to the problem, three robust critical values are proposed. These rely on the actual finite sample distribution of tests and are shown to perform well in simulations and applications.
Work Package 4: The work carried out on the simulation study has been more extensive than initially planned. Comprehensive Monte Carlo simulation designs corresponding to different strengths of identification analyze numerically the finite sample properties of estimators and tests in several models that belong to the class of models under study. The models which were studied in detail are: STAR, GARCH, ARMA, MIDAS. As expected the impact of identification loss is clearly shown by the Monte Carlo simulations.
Work Package 5: Two empirical applications illustrate the contrasting results that emerge from using the standard critical values and the robust critical values. The first application focuses on the GARCH model and uses data on the S&P500 index from January 1947 to September 1984. The second application focuses on the MIDAS model and uses data on macroeconomic indicators downloaded from the Archival Federal Reserve Economic Data (ALFRED) maintained by the Federal Reserve Bank of St. Louis. The empirical applications draw attention to the misleading inference results that can appear when working with models affected by identification issues.
Work Package 6: The results from the previous working package have been summarized for the most part.
Several exploitation and dissemination activities have been undertaken. Here I mention some of the most important ones:
-Research Seminar at De Nederlandsche Bank (DNB)
-Asian Meeting of the Econometrics Society
-Annual Conference of the Romanian Academic Economists from Abroad (ERMAS)
-Barcelona GSE Summer Forum
-Netherlands Econometrics Conference (NESG)
Other outreach activities undertaken to disseminate the projects results were:
-Invited Lecturer, at the Bucharest University of Economic Studies, Summer School
-Research disseminated at the Sustainable Development Goals meeting, EUR, pitch talk
-Research disseminated at the Erasmus School of Economics Female Researchers’ Network
-Research disseminated to the Bachelor Honors Research Class (BHRC) program, EUR
Work Package 1: The most recent related literature has been surveyed thoroughly and linked to the project. Similarities and differences with respect to other papers have been summarized and clarified.
Work Package 2: The asymptotic distribution of predictive ability tests under loss of strong identification has been derived.
Work Package 3: As a solution to the problem, three robust critical values are proposed. These rely on the actual finite sample distribution of tests and are shown to perform well in simulations and applications.
Work Package 4: The work carried out on the simulation study has been more extensive than initially planned. Comprehensive Monte Carlo simulation designs corresponding to different strengths of identification analyze numerically the finite sample properties of estimators and tests in several models that belong to the class of models under study. The models which were studied in detail are: STAR, GARCH, ARMA, MIDAS. As expected the impact of identification loss is clearly shown by the Monte Carlo simulations.
Work Package 5: Two empirical applications illustrate the contrasting results that emerge from using the standard critical values and the robust critical values. The first application focuses on the GARCH model and uses data on the S&P500 index from January 1947 to September 1984. The second application focuses on the MIDAS model and uses data on macroeconomic indicators downloaded from the Archival Federal Reserve Economic Data (ALFRED) maintained by the Federal Reserve Bank of St. Louis. The empirical applications draw attention to the misleading inference results that can appear when working with models affected by identification issues.
Work Package 6: The results from the previous working package have been summarized for the most part.
Several exploitation and dissemination activities have been undertaken. Here I mention some of the most important ones:
-Research Seminar at De Nederlandsche Bank (DNB)
-Asian Meeting of the Econometrics Society
-Annual Conference of the Romanian Academic Economists from Abroad (ERMAS)
-Barcelona GSE Summer Forum
-Netherlands Econometrics Conference (NESG)
Other outreach activities undertaken to disseminate the projects results were:
-Invited Lecturer, at the Bucharest University of Economic Studies, Summer School
-Research disseminated at the Sustainable Development Goals meeting, EUR, pitch talk
-Research disseminated at the Erasmus School of Economics Female Researchers’ Network
-Research disseminated to the Bachelor Honors Research Class (BHRC) program, EUR
The main purpose of the project is to answer the question of how to perform correct forecast comparison when the models are affected by identification deficiencies. To this end, I propose methods that make the standard out-of-sample predictive ability tests robust to this issue.
The carried out work is of interest for the academic research community, but also to central banks and forecasting institutions working with the models under consideration. The latter two are direct potential users of the project results. Communication about the results has been undertaken with: De Nederlandsche Bank (DNB), more formally in a research seminar; and with researchers and personnel working on forecasting tasks at the National Bank of Romania, via one-on-one discussions at the Annual Conference of the Romanian Academic Economists from Abroad. They could then take-up the new knowledge for better informed policy decisions.
In terms of wider societal implication of the project so far, I would mention the impact that has also been made on students by igniting their interest for academic research in my fields. The research has been presented to the Bachelor Honors Research Class (BHRC) program which has the role of encouraging students to pursue academic careers. In addition, I inspired and supervised students for their Bsc/MSc theses and helped them with their applications for their next degrees. Many of them obtained great placements and continued their studies at Oxford, Yale, Southern University of Denmark and Boston College.
The carried out work is of interest for the academic research community, but also to central banks and forecasting institutions working with the models under consideration. The latter two are direct potential users of the project results. Communication about the results has been undertaken with: De Nederlandsche Bank (DNB), more formally in a research seminar; and with researchers and personnel working on forecasting tasks at the National Bank of Romania, via one-on-one discussions at the Annual Conference of the Romanian Academic Economists from Abroad. They could then take-up the new knowledge for better informed policy decisions.
In terms of wider societal implication of the project so far, I would mention the impact that has also been made on students by igniting their interest for academic research in my fields. The research has been presented to the Bachelor Honors Research Class (BHRC) program which has the role of encouraging students to pursue academic careers. In addition, I inspired and supervised students for their Bsc/MSc theses and helped them with their applications for their next degrees. Many of them obtained great placements and continued their studies at Oxford, Yale, Southern University of Denmark and Boston College.