Title: Optimization in Scientific ML - is Accuracy Certification a Viable Goal?
Abstract: Generating nonlinear reduced models for parametric families of PDEs or learning associated parameter-to-solution maps often leads to minimizing a convex energy in a Hilbert space over a nonlinear hypothesis class. Hence, optimization over the budget of trainable weights, defining the hypothesis class, becomes (highly) non-convex. In this talk we discuss adaptive strategies for enhancing robustness and accuracy of optimization success, revolving around "conditional convergence guarantees". The main conceptual constituents are natural gradient flow in combination with an adaptive network expansion. Adaptation aims at increasing alignment with an ideal descent path determined by the gradient in the ambient (infinite dimensional) Hilbert space. This can be further combined with perturbed proximal point methods that can be understood as approximate implicit Euler schemes for the ideal Hilbert space gradient flow. The rationale is that each proximal point step is (much) easier to realize than the global optimization task because in each step the initial guess gets the closer to the optimum, the smaller the step-size. This adaptation to an implicit ideal Hilbert space gradient directions is of interest because the dynamical systems arising in the context of natural gradient flows are generically extremely stiff or even DAEs. Exploiting stable variational formulations of the PDE system, allows one to derive in certain relevant cases a posteriori criteria to check whether an approximate proximal step is sufficiently accurate to guarantee convergence. Roughly, network expansions are triggered when these criteria fail to be met even when the step-size reaches some lower bound. In this sense the results can be understood as "conditional convergence" with accuracy certificates. Part of the material draws on the recent paper [1].
Bio: Wolfgang Dahmen is a Research Professor in the Department of Mathematics at the University of South Carolina, where he previously held the SmartState Chair and Williams-Hedberg-Hedberg Chair. He earned his Ph.D. from RWTH Aachen and his Habilitation from the University of Bonn and has held faculty appointments at the University of Bielefeld, the Free University of Berlin, and RWTH Aachen. His research spans applied and numerical analysis, approximation theory, model order reduction, high-dimensional problems, and scientific machine learning. Dr. Dahmen is a recipient of the Gottfried Wilhelm Leibniz Prize, the Robert Piloty Prize, and the Keck Futures Initiative Award of the US National Academies. He is an elected Fellow of both SIAM and the AMS, a member of the German National Academy of Sciences Leopoldina and the North Rhine-Westphalian Academy of Sciences, and former Chair of the Foundations of Computational Mathematics (FoCM).
]]>Title: Optimization in Scientific ML - is Accuracy Certification a Viable Goal?
]]>Title: Bridging Particle-scale Physics to Continuum Closure for Particle-laden Flows
Title: Bridging Particle-scale Physics to Continuum Closure for Particle-laden Flows
]]>Title: Applying Scientific Computing Perspectives to Neural Network Training
Abstract: Machine learning techniques and their AI decedents are increasingly important part of modern science and engineering. They have proven to be powerful in areas such as surrogate model development, hypothesis generation, and data analysis. Often, these approaches rely on a parameterized function approximation known as the neural network. These approximators are renowned for their flexibility and relative ease of use (if not accuracy). With that ease of use comes a large computational expense. For instance, recent work has demonstrated that pre-training a LLM on the Frontier supercomputer would require two years with ideal parallelization [1]. This talk seeks to examine the neural network with a scientific computing lens to enhance our ability to understand details of the approximation and improve training. Underpinning these ideas is an aspirational hypothesis that insights will lead to multilevel (or hierarchical) methodologies to accelerate training using dramatically fewer resources.
This talk proceeds in two parts. In the first part, we discuss an adaptive basis perspective that has proved fruitful in Scientific Machine Learning (SciML) [2]. The central idea is that the output of the penultimate layer can be seen as providing a basis for the approximation space. With this perspective we develop efficient “operator-split” training algorithms, and new initialization strategies motivated by stability concerns. The second part extends the adaptive basis perspective by considering all layers of the neural network to be basis functions in a learned approximation space. We discuss the challenges associated with the typical neural network architectures that use globally supported basis functions. We then describe Kolmogorov-Arnold networks (KANs) as an alternate architecture built on locally supported splines basis functions [3]. A one-to-one relationship between KANs and a multi-channel ReLU network is shown and used to argue a relaxation property reminiscent of properties of matrix-split iterative linear solvers. Finally, using a natural refinement algorithm combined with the relaxation property, we develop a multilevel training algorithm for KANs demonstrating it on problems in SciML.
Bio: Eric C. Cyr has been at Sandia Albuquerque since 2009 and is a Distinguished Member of the Technical staff. Prior to that he received a BS in Computer Science from Clemson University in 2002, and a PhD in Computer Science from University of Illinois at Urbana-Champaign in 2008. His formal training is in numerical methods, and scientific computing. As a Sandian he has worked in a range of diverse areas including high-performance computing, preconditioners for multi-physics, discretizations for computational plasma simulation, sensitivity analysis for PDEs, numerical optimization, and software development for science applications. In 2018, Eric received the DOE Early Career Award to fund a focus on developing layer-parallel methods for training deep neural networks. This work has broadened his exposure to machine learning. To this end, Eric is working to extend the applicability of HPC to deep learning, understand the behavior and approximation properties of neural networks, and develop methodologies where machine learning can be used to enhance the impact of computation on science and engineering disciplines.
]]>Title: Applying Scientific Computing Perspectives to Neural Network Training
]]>