Learning Orthogonal Multi-Index Models BeyondSmall Initialization: Incremental Learning, Competitive Dynamics and Symmetry

Abstract: Recent work has identified incremental learning in shallow networks trained on single-index and multi-index models. However, existing analyses often rely on simplifying settings, such as small initialization, correlation loss, or layer-wise training. These choices reduce neuron interactions and leave some feature learning dynamics under standard initialization unexplored. We study training dynamics for polynomial-width two-layer networks learning orthogonal multi-index targets under standard initialization using polynomially many samples. We first prove that incremental learning still occurs: the loss decreases sequentially according to the Hermite expansion of the target, with lower-order components learned before higher-order components recover the individual target directions. In this standard initialization regime, training also shows a competitive reallocation of parameter mass: after the total mass fits the target mean and stabilizes, mass shifts into the target subspace and then concentrates on aligned neurons. Our theoretical analysis uses slightly modified gradient flow, while vanilla gradient descent empirically exhibits the same qualitative dynamics. Technically, we introduce a symmetry-based finite-width approximation via symmetrized networks, rather than comparing directly with an infinite-width limit. This yields better control of approximation errors and may be of independent interest.
 

Bio: Mo Zhou is currently a postdoctoral researcher at the Paul G. Allen School of Computer Science & Engineering at the University of Washington, working with Simon Du and Maryam Fazel. He received his Ph.D. from Duke University, where he was advised by Rong Ge. His research focuses on the mathematical foundations of machine learning, particularly the training dynamics and optimization of neural networks.