The basic aim of this project is to suggest novel estimation methodologies for high dimensional datasets. More specifically, we aspire to propose a general framework that permits estimation of links, or connections, across an increasingly large set of variables (economic or not), which are non-constant, across time. To this end, we address two realistic features of observed datasets which have been barely tackled together in the literature, so far. These are the time varying structure and the large dimensionality of economic datasets.
Time variation in economic relationships has been largely studied in economics. It can be seen either as abrupt shifts in the assumed generating mechanisms of the variables, or as smooth stochastic or deterministic changes in that. Either way, it can be considered as the result of altering forces such as institutional switching, economic transitions, preference fluctuations, policy transformations or technological changes, inter alia. All these can imply instabilities in the assumed economic relationships.
Large datasets are, nowadays, a key characteristic of human development (e.g. computers, being in the middle of most economic transactions generate huge amounts of data that can be analyzed to extract critical information). This is relevant for answering economic policy questions or a key to various scientific discoveries. In large datasets, conventional statistical and econometric techniques such as sample covariance estimation or regression coefficient estimation fail to work consistently due to the dimensionality of the estimation object. For instance, in a linear economic relationship we frequently obtain T observations of a dependent variable (y) as a function of many potential predictors (p predictors). When the number of predictors p is large or larger than the temporal dimension T, then a regression with all available covariates becomes extremely problematic if not impossible. Analogously, when our aim is to estimate the large covariance matrix of the p predictors, the sample estimate becomes heavily unreliable. It is also, particularly, computationally demanding since the dimension of the estimated object rises as a square of the dimension of the dataset under analysis. The current literature provides some novel answers but only when we assume a fixed, across time, covariance matrix, of the true data generating mechanism.
These two aspects of the observed datasets are important characteristics of the reality and failure to provide a framework that can accommodate these, simultaneously, will certainly result to unreliable scientific discoveries. In economics, this implies that the developed models will be insufficient to capture important characteristics of the economy, delivering false or unsuccessful policy suggestions.
We provide a unified framework that can accommodate these aspects of real datasets, with nice theoretical properties. To this end, the large dimensional econometrics literature, is combined with the non parametric estimation literature, in an innovative fashion, and novel methodologies on large covariance matrix and large dimensional regression, are proposed. As it is shown, our methods imply significant improvements, in a wide range of applications and metrics, over the relevant methodologies that currently dominate the literature.
