WP1:
Task 1 studies the fundamental trade-offs in primal dual optimization, resulting in new algorithms that achieve unparalleled scaling. Our analysis relies on a novel combination of four ideas applied to the primal-dual gap function: smoothing, acceleration, homotopy, and coordinate descent with non-uniform sampling. As described in Tasks 2 and 4, we study different notions of smoothness and developed an extension of the famous mirror descent methods that only require differentiability. We then focused on the structure of the constraints as promised by Task 3 to also resolve an open problem in sampling via the Langevin dynamics. Furthermore, we developed a primal-dual method to handle infinitely many constraints.
WP2:
Task 1 develops a novel method for convex unconstrained optimization that, without any modifications, ensures: (i) accelerated convergence rate for smooth objectives, (ii) standard convergence rate in the general (non-smooth) setting, and (iii) standard convergence rate in the stochastic optimization setting. Task 2 derives a non-Euclidean optimization framework in the non-convex setting that takes nonlinear gradient steps on the matrix factorization problems. We also developed a conditional gradient framework for a composite convex minimization template with broad applications, including semidefinite programming. We have also developed new algorithms that can solve convex optimization problems in space required to specify the problem and its solution. For scalability questions in Task 4, we initially took a sketching based approach. As a result, we obtain optimal rates for regularized algorithms with randomized sketches, provided that the sketch dimension is proportional to the effective dimension up to a logarithmic factor. This particular result also relates to WP3 below.
WP3:
Task1 focuses on the fundamental effects of regularized learning formulations, where we investigate a class of spectral/regularized algorithms, including ridge regression, principal component regression, and gradient methods. We then obtained some new results for Tasks 2 and 3 along with non-linear models with group testing. In addition, we looked into performing maximizing an unknown function that has a particular graph structure that we can exploit. This approach enables us to exploit the underlying structures in non-linear and non-convex models and perform statistical inference tasks, even applying to general automatic machine learning tuning tasks.
WP4:
As promised in Task 1, we developed a sampling design framework for non-linear decoders and have managed to demonstrate it with real magnetic resonance imaging data along with statistical generalization guarantees. These results are the first of its kind, in handling discrete optimization problems with nearly submodular structure, not only in the deterministic setting but also handling the stochastic setting where there is a clear memory-performance trade-off. We have also obtained new sketching results that support solutions of semidefinite programming in small space.
WP5:
Task 1 develops a test bed for a super resolution imaging system and have obtained preliminary sampling results. For Task 2, we have managed to collaborate with the material scientists and managed to publish at a chemistry journal. For WP5, we have also obtained strong initial demonstration results for data sampling. In particular, our trained sampling operators provide great area, power, and performance trade-offs for neural signal acquisition as automating machine learning systems and have managed to collaborate with MRI engineers to demonstrate results with real data.
