{"681481":{"#nid":"681481","#data":{"type":"event","title":"PhD Defense by Zhongzhi Yu","body":[{"value":"\u003Cp\u003ETitle: Enhancing Foundation Models with Self-Guided Techniques: From Attention to Adapters to Agents\u003C\/p\u003E\u003Cp\u003EDate: Thursday, April 10th\u003Cbr\u003ETime: 10:00 AM \u2013 11:30 AM (Eastern Time)\u003Cbr\u003ELocation: Klaus 1212, Klaus Advanced Computing Building\u003Cbr\u003EZoom Link: https:\/\/gatech.zoom.us\/j\/9960405372\u003C\/p\u003E\u003Cp\u003EZhongzhi Yu\u003Cbr\u003EPh.D. Student\u003Cbr\u003ESchool of Computer Science\u003Cbr\u003EGeorgia Institute of Technology\u003C\/p\u003E\u003Cp\u003ECommittee:\u003Cbr\u003EDr. Yingyan (Celine) Lin (Advisor, School of Computer Science, Georgia Tech)\u003Cbr\u003EDr. Chao Zhang (School of Computational Science \u0026amp; Engineering, Georgia Tech)\u003Cbr\u003EDr. Haoxing (Mark) Ren (Nvidia Corporation)\u003Cbr\u003EDr. Pavlo Molchanov (Nvidia Corporation)\u003Cbr\u003EDr. Zsolt Kira (School of Interactive Computing, Georgia Tech)\u003C\/p\u003E\u003Cp\u003EAbstract:\u003C\/p\u003E\u003Cp\u003EFoundation models, a class of large-scale transformers pretrained on large-scale datasets, have achieved remarkable performance across various applications. However, the growing demand to deploy foundation models in real-world applications with diverse resource and capability requirements highlights three critical challenges hindering their broader adoption: (1) the accuracy-efficiency trade-off, where improving accuracy through scaling leads to prohibitive computational costs; (2) inefficient adaptation strategies that require heavy supervision and resources, hindering use in resource-constrained environments; and (3) limited capabilities in handling complex tasks, such as automated hardware code generation and multi-agent collaboration.\u003C\/p\u003E\u003Cp\u003EThis thesis addresses these challenges by leveraging our insight that foundation models encode rich representations, which, if effectively extracted, can enable self-guided optimization. Specifically, we introduce a set of techniques across three complementary levels, each targeting one of the aforementioned challenges: (1) At the attention level, addressing the accuracy-efficiency trade-off, we introduce the Attention Calibration Technique (ACT), which refines suboptimal attention distributions to improve performance without training, and SpotVLM, which reduces visual token redundancy in video-language models through attention-based selection. (2) At the adapter level, targeting adaptation efficiency, we present Master-ASR, which enables dynamic selection and composition of adapters to support efficient model adaptation. (3) At the agent level, targeting complex tasks that require knowledge retrieval and reasoning, we propose Instant-RAG, a retrieval-augmented generation system that hides retrieval overhead within the standard generation workflow to enable efficient knowledge access, and Spec2RTL-Agent, which addresses the challenging task of directly generating Register Transfer Level (RTL) code from specification documents by coordinating multiple foundation models to achieve advanced reasoning capabilities. Together, these techniques form a comprehensive framework for self-guided optimization that addresses key challenges limiting the broader deployment of foundation models, enabling more accessible and capable models in real-world scenarios.\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E","summary":"","format":"limited_html"}],"field_subtitle":"","field_summary":[{"value":"\u003Cp\u003EEnhancing Foundation Models with Self-Guided Techniques: From Attention to Adapters to Agents\u003C\/p\u003E","format":"limited_html"}],"field_summary_sentence":[{"value":"Enhancing Foundation Models with Self-Guided Techniques: From Attention to Adapters to Agents"}],"uid":"27707","created_gmt":"2025-03-31 19:05:35","changed_gmt":"2025-03-31 19:18:16","author":"Tatianna Richardson","boilerplate_text":"","field_publication":"","field_article_url":"","field_event_time":{"event_time_start":"2025-04-10T10:00:00-04:00","event_time_end":"2025-04-10T12:00:00-04:00","event_time_end_last":"2025-04-10T12:00:00-04:00","gmt_time_start":"2025-04-10 14:00:00","gmt_time_end":"2025-04-10 16:00:00","gmt_time_end_last":"2025-04-10 16:00:00","rrule":null,"timezone":"America\/New_York"},"location":"Klaus 1212, Klaus Advanced Computing Building","extras":[],"groups":[{"id":"221981","name":"Graduate Studies"}],"categories":[],"keywords":[{"id":"100811","name":"Phd Defense"}],"core_research_areas":[],"news_room_topics":[],"event_categories":[{"id":"1788","name":"Other\/Miscellaneous"}],"invited_audience":[{"id":"78771","name":"Public"}],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[],"email":[],"slides":[],"orientation":[],"userdata":""}}}