Title: Feynman-Kac Numerical Techniques for Stochastic Optimal Control

Date: Monday, August 9th, 2021

Time: 11:00 AM - 1:00 PM (EDT)

Location: BlueJeans meeting (https://bluejeans.com/303268392/2685)

Kelsey Hawkins

Robotics Ph.D. Candidate

Institute for Robotics and Intelligent Machines

Georgia Institute of Technology

Committee:

Dr. Panos Tsiotras (Advisor) – School of Aerospace Engineering, Georgia Tech

Dr. Dmitry Berenson – Electrical Engineering and Computer Science Dept., University of Michigan

Dr. Sam Coogan – School of Electrical and Computer Engineering, Georgia Tech

Dr. Evangelos Theodorou – School of Aerospace Engineering, College of Computing, Georgia Tech

Dr. Kyriakos Vamvoudakis – School of Aerospace Engineering, Georgia Tech

Abstract:

Three significant advancements are proposed for improving numerical methods in the solution of forward-backward stochastic differential equations (FBSDEs) appearing in the Feynman-Kac representation of the value function in stochastic optimal control (SOC) problems. First, we propose a novel characterization of FBSDE estimators as either on-policy or off-policy, highlighting the intuition for these techniques that the distribution over which value functions are approximated should, to some extent, match the distribution the policies generate.

Second, two novel numerical estimators are proposed for improving the accuracy of single-timestep updates. In the case of LQR problems, we demonstrate both in theory and in numerical simulation that our estimators result in near machine-precision level accuracy, in contrast to previously proposed methods that can potentially diverge on the same problems.

Third, we propose a new method for accelerating the global convergence of FBSDE methods. By the repeated use of the Girsanov change of probability measures, it is demonstrated how a McKean-Markov branched sampling method can be utilized for the forward integration pass, as long as the controlled drift terms are appropriately compensated in the backward integration pass. Subsequently, a numerical approximation of the value function is proposed by solving a series of function approximation problems backwards in time along the edges of a space-filling tree.