{"675239":{"#nid":"675239","#data":{"type":"event","title":"Phd Defense by Arjun Majumdar","body":[{"value":"\u003Cp\u003E\u003Cstrong\u003ETitle\u003C\/strong\u003E: Large-Scale Offline Pre-Training Bootstraps Embodied Intelligence\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EArjun Majumdar\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EPh.D. Student in Computer Science\u003C\/p\u003E\u003Cp\u003ESchool of Interactive Computing\u003C\/p\u003E\u003Cp\u003EGeorgia Institute of Technology\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EDate:\u003C\/strong\u003E July 11th, 2024\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003ETime:\u003C\/strong\u003E 12:00 - 1:30pm EST \/ 9:00 - 10:30 am PST\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003ELocation:\u003C\/strong\u003E \u003Ca href=\u0022https:\/\/gatech.zoom.us\/j\/96912711656?pwd=7yOzgg4vafrSbOGKdrgyTVwaZQGxde.1\u0026amp;from=addon\u0022\u003Ezoom\u003C\/a\u003E\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003ECommittee\u003C\/strong\u003E:\u003C\/p\u003E\u003Cp\u003EDr. Dhruv Batra (Advisor) -- School of Interactive Computing, Georgia Institute of Technology\u003C\/p\u003E\u003Cp\u003EDr. Zsolt Kira -- School of Interactive Computing, Georgia Institute of Technology\u003C\/p\u003E\u003Cp\u003EDr. James Hays -- School of Interactive Computing, Georgia Institute of Technology\u003C\/p\u003E\u003Cp\u003EDr. Jitendra Malik -- University of California Berkeley\u003C\/p\u003E\u003Cp\u003EDr. Vincent Vanhoucke \u2013- Google DeepMind\u003C\/p\u003E\u003Cp\u003EDr. Vladlen Koltun -- Apple\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EAbstract:\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EA central goal in Artificial Intelligence is building embodied agents (such as mobile robots) that are generalists -- capable of assisting with a wide-variety of tasks (specified in natural language) in any environment or setting. Such agents must understand a vast diversity of concepts in the visual world and be able to ground (or associate) this understanding with language to allow users to describe tasks and goals. How can we develop agents with such an extensive and functional understanding of the world?\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003EIn this thesis, we will argue that offline pre-training of foundation models on web-scale data enables embodied intelligence. In part 1, we present VC-1, a visual foundation model pre-trained (primarily) on video data collected from an egocentric perspective. We systematically demonstrate that such a model substantially benefits from increasing pre-training dataset diversity by introducing CortexBench, an embodied AI (EAI) benchmark curated from a diverse collection of existing EAI tasks (requiring locomotion, navigation, and dexterous and mobile manipulation of objects). In part 2, we first demonstrate that visual grounding learned from internet data (i.e., image-caption pairs from the web) can be transferred to an instruction-following visual navigation agent (VLN-BERT). Then, we present ZSON, a highly scalable approach for learning to visually navigate to objects specified in open-vocabulary, natural language instructions such as \u201cfind the kitchen sink.\u201d The key idea is to leverage a pre-trained visiolinguistic embedding space (from CLIP) to decouple learning to represent semantic goals (such as a \u201ca kitchen sink\u201d) from learning to navigate to semantic goals. In part 3, we present a modern formulation of the Embodied Question Answering (EQA) task, which requires understanding a 3D environment well enough to answer questions about it in natural language. We introduce a new benchmark (OpenEQA) and study a modular agent that leverages pre-trained components such as vision-language models (VLMs) to address the EQA task.\u003C\/p\u003E","summary":"","format":"limited_html"}],"field_subtitle":"","field_summary":[{"value":"\u003Cp\u003ELarge-Scale Offline Pre-Training Bootstraps Embodied Intelligence\u003C\/p\u003E","format":"limited_html"}],"field_summary_sentence":[{"value":"Large-Scale Offline Pre-Training Bootstraps Embodied Intelligence"}],"uid":"27707","created_gmt":"2024-06-26 15:29:51","changed_gmt":"2024-06-26 15:30:30","author":"Tatianna Richardson","boilerplate_text":"","field_publication":"","field_article_url":"","field_event_time":{"event_time_start":"2024-07-11T12:00:00-04:00","event_time_end":"2024-07-11T13:00:00-04:00","event_time_end_last":"2024-07-11T13:00:00-04:00","gmt_time_start":"2024-07-11 16:00:00","gmt_time_end":"2024-07-11 17:00:00","gmt_time_end_last":"2024-07-11 17:00:00","rrule":null,"timezone":"America\/New_York"},"location":"ZOOM","extras":[],"groups":[{"id":"221981","name":"Graduate Studies"}],"categories":[],"keywords":[{"id":"100811","name":"Phd Defense"}],"core_research_areas":[],"news_room_topics":[],"event_categories":[{"id":"1788","name":"Other\/Miscellaneous"}],"invited_audience":[{"id":"78771","name":"Public"},{"id":"78761","name":"Faculty\/Staff"}],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[],"email":[],"slides":[],"orientation":[],"userdata":""}}}