To achieve our goal, our main approach is to view the problem from a "dynamical systems" perspective. This perspective is natural, as the optimization algorithms that are used in deep learning systems are predominantly iterative, which constitute a dynamical system that evolves over time. From this perspective, we have so far analyzed learning algorithms from two points of view:
1) Heavy-tail properties: Heavy-tailed distributions likely produce observations that can be very large in magnitude and far from the mean; hence, they are often used for modeling phenomena that exhibit outliers. Despite their ‘daunting’ connotation, heavy tails are ubiquitous in virtually any domain: In the context of machine learning, recent studies (some of which have been co-authored by the PI) have shown that heavy tails naturally emerge in various ways, and, contrary to their perceived image, they can be in fact beneficial for the performance.
In this first direction, we rigorously analyzed the emergence and the impact of heavy tails in stochastic optimization. Our results have revealed several surprising phenomena:
• We investigated the emerging mechanism of heavy tails in stochastic optimization. We showed that certain popularly used heuristics in “online” stochastic optimization (e.g. cyclic step-sizes) have a direct impact on the tails of the parameters that are produced by the optimization algorithm. We then focused on the more realistic case “offline” case where the optimizer only has access to a finitely many training points. We showed that, in this case the iterates will have “approximate” heavy-tails, meaning that as the number of data points increases, the iterates will exhibit heavy tails. These two results shed more light on why and how heavy tails emerge in stochastic optimization.
• We proved for the first time that the relation between heavy tails and generalization error is not monotonic: depending on several factors (that we explicitly identified), heavy tails can either harm or improve generalization. This has clarified the general picture and brought new insights about algorithmic design.
2) In the second direction, we focused on the “geometry” of machine learning algorithms. We extended the state-of-the-art in terms of understanding how the topological properties of stochastic optimization algorithms affect the generalization error. In particular, we have proved novel error bounds for both continuous-time and discrete-time optimization algorithms. During this process, we discovered novel topological constructs for optimizers for the first time, and rigorously linked them to the generalization error.
These developments led to 16 publications so far, all of them published at top-tier venues such as NeurIPS, ICML, COLT, ALT, and TMLR.
