Practicality meets Optimality: Real-Time Statistical Inference under Complex Constraints
Constrained estimation problems are prevalent in statistics, machine learning, and engineering. These problems encompass constrained generalized linear models, constrained deep neural networks, physics-inspired machine learning, algorithmic fairness, and optimal control. However, existing estimation methods under hard constraints rely on either projection or regularization, which may theoretically exhibit optimal efficiency but are impractical or unreasonably fail in reality. This talk aims to bridge the significant gap between practice and theory for constrained estimation problems.
I will begin by introducing the critical methodology used to bridge the gap, called Stochastic Sequential Quadratic Programming. We will see that SQP methods serve as the workhorse for modern scientific machine learning problems and can resolve the failure modes of prevalent regularization-based methods. I will demonstrate how to make SQP adaptive and scalable using various modern techniques, such as stochastic line search, trust region, and dimension reduction.
Additionally, I will show how to further enhance SQP to handle inequality constraints online.
Following the methodology, I will present some selective theories, emphasizing the consistency and efficiency
of the SQP methods. Specifically, I will show that online SQP iterates asymptotically exhibit normal behavior
with a mean of zero and optimal covariance in the Hájek and Le Cam sense. Significantly, the covariance does
not deteriorate even when we apply modern techniques driven by practical concerns. The talk concludes with
experiments on both synthetic and real datasets.
Sen Na is currently a postdoctoral researcher in the Department of Statistics and the International Computer
Science Institute at UC Berkeley. He received a Ph.D. degree in statistics from the University of Chicago.
Sen Na’s primary research interests lie in the mathematical foundations of data science, encompassing high dimensional statistics, computational statistics, sequential decision-making, and large-scale and stochastic
nonlinear optimization. Additionally, he is passionate about various applications of machine learning methods in scientific fields such as biology, neuroscience, physics, and engineering. Sen Na’s research has been recognized by the prestigious Harper Dissertation Fellowship from UChicago, and he has been selected as one of the Young Researchers in ORIE by Cornell University.
Constrained estimation problems are prevalent in statistics, machine learning, and engineering. These problems
encompass constrained generalized linear models, constrained deep neural networks, physics-inspired machine
learning, algorithmic fairness, and optimal control. However, existing estimation methods under hard constraints
rely on either projection or regularization, which may theoretically exhibit optimal efficiency but are impractical
or unreasonably fail in reality. This talk aims to bridge the significant gap between practice and theory
for constrained estimation problems.
I will begin by introducing the critical methodology used to bridge the gap, called Stochastic Sequential Quadratic
Programming. We will see that SQP methods serve as the workhorse for modern scientific machine learning problems and can resolve the failure modes of prevalent regularization-based methods. I will demonstrate how to make SQP adaptive and scalable using various modern techniques, such as stochastic line search, trust region, and dimension reduction. Additionally, I will show how to further enhance SQP to handle inequality constraints online.
Following the methodology, I will present some selective theories, emphasizing the consistency and efficiency
of the SQP methods. Specifically, I will show that online SQP iterates asymptotically exhibit normal behavior
with a mean of zero and optimal covariance in the Hájek and Le Cam sense. Significantly, the covariance does
not deteriorate even when we apply modern techniques driven by practical concerns. The talk concludes with
experiments on both synthetic and real datasets.