Title: Sim2Robot: Training Robots for the Real-World with Imperfect Simulators

Date: Friday, April 19th, 2024

Time: 3:00 PM – 5:00 PM EST  / 12:00 PM – 2:00 PM PST

Location: Zoom

Joanne Truong

Robotics Ph.D. Student 

School of Interactive Computing 

Georgia Institute of Technology 

Committee: 

Dr. Sonia Chernova (Advisor) – School of Interactive Computing, Georgia Institute of Technology 

Dr. Dhruv Batra (Advisor) – School of Interactive Computing, Georgia Institute of Technology 

Dr. Jie Tan – School of Interactive Computing, Georgia Institute of Technology 

Dr. Vladlen Koltun – Distinguished Scientist, Apple

Dr. Vincent Vanhoucke –  Distinguished Scientist, Google

Dr. Ken Goldberg –  Industrial Engineering and Operations Research Department, University of California Berkeley

Abstract: 

The goal of Artificial Intelligence is to “construct useful intelligent systems”, such as mobile robots, to assist in our day-to-day lives. For these mobile robotic assistants to be useful in the real-world, they must skillfully navigate complex environments (e.g., delivering packages from one building to another). However, training robots in the real-world can be slow, dangerous, expensive, and difficult to reproduce. Thus, one paradigm in robot learning is to leverage simulation for training robots (where gathering experience is scalable, safe, cheap, and reproducible) before being deployed in the real world. However, no simulator is perfect; AI systems learn to “cheat” by exploiting imperfections. Thus, how can we train robots in imperfect simulators while ensuring that the learned skills generalize to reality? 

In this thesis, we will argue that simulators need not be perfect to be useful; they don’t need to model everything about the world, only what’s necessary for generalization. We present 1) Sim2Real Correlation Coefficient (SRCC), a formal approach for simulator design by revealing the degree to which simulation parameters allow agents to cheat. SRCC optimization enables researchers to use simulation for evaluation with confidence that the performance of different models will generalize to real robots. 2) Bi-directional Domain Adaptation (BDA), and Kinematic-to-Dynamic Transfer (Kin2Dyn), sample-efficient methods for reducing the sim2real gap. BDA and Kin2Dyn improve robot learning and generalization to the real-world by utilizing abstracted physics and simple adaptation models learned from small amounts of real-world data. 3) IndoorSim-to-OutdoorReal, an end-to-end learned approach that enables visual navigation in out-of-distribution environments zero-shot. We show that simulators can be used for real-world transfer without having to apriori design and model the deployment scenario. 4) Implicit Map Cross Modal Attention (Implicit MapCMA), a vision and language navigation model that utilizes structured implicit maps for navigating in an environment over time. Agents that leverage implicit maps are able to more effectively follow the ground-truth path indicated by the language instruction.