Title: Embodied Intelligence from Human Experience

Date: Wed Apr 15 3:00-4:30pm

Location: Coda C1315 Grant Park

Zoom: Link

Meeting ID: 242 927 8475

Passcode: 423370

Committee:

Judy Hoffman (Advisor) - School of Information and Computer Science, UC Irvine

Danfei Xu (Advisor) - School of Interactive Computing, Georgia Tech

Siddharth Karamcheti - School of Interactive Computing, Georgia Tech

Dhruv Batra - Chief Scientist, Yutori

Sergey Levine - Department of Electrical Engineering and Computer Sciences, UC Berkeley

Abstract: Robot foundation models have begun to solve dexterous manipulation tasks with sparks of broad open world generalization. The most promising results are driven by large scale teleoperation, but this data source cannot scale as fast as other sources of data, which has spawned considerable work to learn from human videos. Despite progress here, the human to robot gap makes it difficult to directly transfer capabilities in human video to robot policies, forcing most prior works to introduce some inductive bias. In this thesis, we argue that co-design of training algorithms and data collection systems enables us to directly transfer capabilities from human experience. We begin the thesis by proposing embodied human data, defined as egocentric video paired with 3D hand and head pose tracking, as well as a system to collect it, and an algorithm to train with it. Second, we collect the largest dataset of embodied human data, enabling both controlled scientific exploration as well as large scale training. Third, we find that human to robot transfer emerges in foundation models pretrained across many tasks and embodiments. Finally, we propose new work to improve human to robot transfer during pretraining with careful dataset curation and balance.