Global Convergence of Policy Gradient Methods in Structured Markov Decision Processes​

Abstract: Policy gradient methods are widely used in reinforcement learning, yet the nonconvexity of their objectives makes global convergence difficult to establish. We study finite-horizon and infinite-horizon discounted Markov decision processes (MDPs) with general state and action spaces. For each setting, we identify structural conditions under which every stationary point is globally optimal and establish non-asymptotic convergence rates for policy gradient methods. Applications include tabular MDPs, control models such as the linear quadratic regulator, and operations models for inventory and cash management for which we establish the first policy gradient complexity guarantees.
 

Bio: Minda Zhao is a final-year Ph.D. student in OR, advised by Prof. Xin Chen. His research focuses on stochastic programming, nonconvex optimization, and reinforcement learning, with applications to operations management. He studies the structure of optimization problems in queueing and sequential decision-making models, as well as the role of human supervision in algorithmic decision-making.​