Title: Robustness under Distribution Shifts in Computer Vision

Date: Wednesday, June 26, 2024

Time: 2:00 PM – 4:00 PM EST 

Virtual Link: Zoom Link 

Meeting ID: 910 1332 1993

Passcode: 637066

Junjiao Tian

Robotics Ph.D. Student 

School of Electrical and Computer Engineering 

Georgia Institute of Technology  

Committee

Dr. Zsolt Kira (Advisor) – Interactive Computing/Robotics, Georgia Institute of Technology 

Dr. Judy Hoffman – Interactive Computing/Robotics, Georgia Institute of Technology 

Dr. Animesh Garg – Interactive Computing/Robotics, Georgia Institute of Technology 

Dr. Jean Oh – Robotics Institute, Carnegie Mellon University  

Dr. Dustin Tran – DeepMind, Google  

Abstract

In conventional machine learning, the test distribution is often assumed to be the same as the training distribution. However, this assumption has become less justified in the deep learning era. Deep learning models have been increasingly deployed in the real world, in which the distribution of the input data can be very different from that of the training data. For example, a model is more likely to predict instances seen more often in its training data, leading to statistically biased priors (prior shifts); a model can perform drastically worse if the input changes in its appearance slightly and still be very confident in wrong predictions (covariate shifts). In this thesis, we study the causes of deep learning models' suboptimal performance under these two distribution shifts and propose mitigation strategies to improve their robustness to the shifts.

Specifically, optimizing a neural network unconstrainedly can lead to negative unintended consequences, including bias towards large classes if its training data is imbalanced and loss of its original robustness if fined-tuned from a foundation model. This thesis studies how unconstrained optimization negatively affects robustness and proposes constrained optimization strategies to improve robustness to distribution shifts. For prior shifts, we balance the optimization objective during training and calibrate the model to match the desired recall and precision after training. For covariate shifts, we propose efficient and constrained fine-tuning frameworks to fit a model to downstream tasks while maintaining its robustness.