Speaker: Wolfgang Dahmen, professor at the University of South Carolina
Date and Time: October 23, 2:00-3:00 p.m. 
Location: TBD
Host: Peng Chen

Title: Optimization in Scientific ML - is Accuracy Certification a Viable Goal?

Abstract: Generating nonlinear reduced models for parametric families of PDEs or learning associated parameter-to-solution maps often leads to minimizing a convex energy in a Hilbert space over a nonlinear hypothesis class. Hence, optimization over the budget of trainable weights, defining the hypothesis class, becomes (highly) non-convex. In this talk we discuss adaptive strategies for enhancing robustness and accuracy of optimization success, revolving around "conditional convergence guarantees". The main conceptual constituents are natural gradient flow in combination with an adaptive network expansion. Adaptation aims at increasing alignment with an ideal descent path determined by the gradient in the ambient (infinite dimensional) Hilbert space. This can be further combined with perturbed proximal point methods that can be understood as approximate implicit Euler schemes for the ideal Hilbert space gradient flow. The rationale is that each proximal point step is (much) easier to realize than the global optimization task because in each step the initial guess gets the closer to the optimum, the smaller the step-size. This adaptation to an implicit ideal Hilbert space gradient directions is of interest because the dynamical systems arising in the context of natural gradient flows are generically extremely stiff or even DAEs. Exploiting stable variational formulations of the PDE system, allows one to derive in certain relevant cases a posteriori criteria to check whether an approximate proximal step is sufficiently accurate to guarantee convergence. Roughly, network expansions are triggered when these criteria fail to be met even when the step-size reaches some lower bound. In this sense the results can be understood as "conditional convergence" with accuracy certificates. Part of the material draws on the recent paper [1].

Bio: Wolfgang Dahmen is a Research Professor in the Department of Mathematics at the University of South Carolina, where he previously held the SmartState Chair and Williams-Hedberg-Hedberg Chair. He earned his Ph.D. from RWTH Aachen and his Habilitation from the University of Bonn and has held faculty appointments at the University of Bielefeld, the Free University of Berlin, and RWTH Aachen. His research spans applied and numerical analysis, approximation theory, model order reduction, high-dimensional problems, and scientific machine learning. Dr. Dahmen is a recipient of the Gottfried Wilhelm Leibniz Prize, the Robert Piloty Prize, and the Keck Futures Initiative Award of the US National Academies. He is an elected Fellow of both SIAM and the AMS, a member of the German National Academy of Sciences Leopoldina and the North Rhine-Westphalian Academy of Sciences, and former Chair of the Foundations of Computational Mathematics (FoCM).