Name: Yifan Lin

Title: Data-driven Stochastic Optimization in the Presence of Distributional Uncertainty

Advisor:

Dr. Enlu Zhou, Industrial and Systems Engineering, Georgia Tech

Committee Members:

Dr. Alexander Shapiro, Industrial and Systems Engineering, Georgia Tech

Dr. Siva Theja Maguluri, Industrial and Systems Engineering, Georgia Tech

Dr. Guanghui Lan, Industrial and Systems Engineering, Georgia Tech

Dr. Fumin Zhang, Electrical and Computer Engineering, Hong Kong University of Science and Technology, Georgia Tech (adjunct)

Time: Monday, April 8th at 9:00 AM ET

Location: Remote

Meeting Link (Teams): 

https://teams.microsoft.com/l/meetup-join/19%3anijI4DWy5o8DYh-ARd3HCuOlXxKf90lZorw6hrLdQfU1%40thread.tacv2/1706971881656?context=%7b%22Tid%22%3a%22482198bb-ae7b-4b25-8b7a-6d7f32faa083%22%2c%22Oid%22%3a%220c907ddf-6703-45d0-9902-0bfabd39eeb6%22%7d

Meeting ID: 296 309 389 312

Passcode: M3H7jG

Abstract:
Stochastic optimization is a mathematical framework that models decision making under uncertainty. It usually assumes that the decision maker has full knowledge about the underlying uncertainty through a known probability distribution and minimizes (or maximizes) a functional of the cost (or reward) function. However, the probability distribution of the randomness in the system is rarely known in practice and is often estimated from historical data. The goal of the decision maker is therefore to select the optimal decision under this distributional uncertainty. This thesis aims to address the distributional uncertainty in the context of stochastic optimization by proposing new formulations and devising new approaches.

In Chapter 2, we consider stochastic optimization under distributional uncertainty, where the unknown distributional parameter is estimated from streaming data that arrive sequentially over time. Moreover, data may depend on the decision of the time when they are generated. For both decision-independent and decision-dependent uncertainties, we propose an approach to jointly estimate the distributional parameter via Bayesian posterior distribution and update the decision by applying stochastic gradient descent on the Bayesian average of the objective function.

In Chapter 3, we deviate from the static stochastic optimization studied in the previous chapter and instead focus on a multistage setting. Specifically, we consider a class of sequential decision making problem called multi-armed bandit (MAB). In certain situations, the decision maker may also be provided with contexts. We consider the contextual MAB with linear payoffs under a risk-averse criterion. In particular, we consider mean-variance as the risk criterion, and the best arm is the one with the largest mean-variance reward. We apply the Thompson Sampling algorithm and provide a comprehensive regret analysis for a variant of the proposed algorithm.

In Chapter 4, we consider the multistage stochastic optimization problem in the context of Markov decision processes (MDPs). We propose a new formulation, Bayesian risk Markov Decision Process (BR-MDP), to address distributional uncertainty in MDPs, where a risk functional is applied in nested form to the expected total cost with respect to the Bayesian posterior distributions of the unknown parameters. The proposed formulation provides more flexible risk attitudes towards distributional uncertainty and takes into account the availability of data in future time stages.

In Chapter 5, we consider a more general reinforcement learning setting, and focus on improving the sample efficiency of policy optimization algorithm. We study the natural policy gradient method with reusing historical trajectories via importance sampling. We show that the resultant algorithm is convergent, and reusing helps improve the convergence rate.