{"675584":{"#nid":"675584","#data":{"type":"event","title":"PhD Proposal by Zhongzhi Yu","body":[{"value":"\u003Cp\u003E\u003Cstrong\u003ETitle: Improving Large-Scale Foundation Models Via Attention-Aware Techniques\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003E\u0026nbsp;\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EDate:\u0026nbsp;\u003C\/strong\u003EThursday, August 1st, 2024\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003ETime:\u0026nbsp;\u003C\/strong\u003E2:00 pm - 3:00 pm ET\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003ELocation:\u0026nbsp;\u003C\/strong\u003EVirtually via Zoom (\u003Ca href=\u0022https:\/\/gatech.zoom.us\/j\/9960405372?pwd=bzhIbVdWRkxweW9naUh0aUt4ci9WZz09\u0022 target=\u0022_blank\u0022\u003Ehttps:\/\/gatech.zoom.us\/j\/9960405372?pwd=bzhIbVdWRkxweW9naUh0aUt4ci9WZz09\u003C\/a\u003E)\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003E\u0026nbsp;\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EPhD Student:\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EZhongzhi Yu, School of Computer Science, Georgia Institute of Technology\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003E\u0026nbsp;\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003ECommittee Members:\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EDr. Yingyan (Celine) Lin (Advisor) \u2013 School of Computer Science, Georgia Institute of Technology\u003C\/p\u003E\u003Cp\u003EDr. Chao Zhang\u0026nbsp;\u2013 School of Computational Science and Engineering, Georgia Institute of Technology\u003C\/p\u003E\u003Cp\u003EDr. Judy Hoffman\u0026nbsp;\u2013 School of Interactive Computing, Georgia Institute of Technology\u003C\/p\u003E\u003Cp\u003EDr. Pavlo Molchanov\u0026nbsp;\u2013 Nvidia Corporation\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003E\u0026nbsp;\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EAbstract:\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EFoundation models, which are a series of large-scale transformer models, have shown impressive performance across a diverse range of applications, from natural language processing to computer vision. The key enabler behind their success is the attention module, which controls how these models extract relationships among input tokens. However, despite the importance of the attention module, our understanding of its role during the inference and fine-tuning stages remains limited, leading to challenges such as potentially sub-optimal model performance and a lack of interpretability.\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003EMy thesis research focuses on understanding the potentially suboptimal attention distributions generated by foundation models and developing attention-aware techniques to improve their performance. The primary insight from my research is that certain high-attention tokens can negatively affect foundation model performance during both fine-tuning and inference. Building on this insight, my research presents state-of-the-art solutions to enhance the performance of foundation models, including an attention-aware data augmentation technique that enhances the data efficiency of the fine-tuning process and an attention calibration technique that improves inference accuracy.\u003C\/p\u003E","summary":"","format":"limited_html"}],"field_subtitle":"","field_summary":[{"value":"\u003Cp\u003E\u003Cstrong\u003EImproving Large-Scale Foundation Models Via Attention-Aware Techniques\u003C\/strong\u003E\u003C\/p\u003E","format":"limited_html"}],"field_summary_sentence":[{"value":"Improving Large-Scale Foundation Models Via Attention-Aware Techniques"}],"uid":"27707","created_gmt":"2024-07-24 19:50:18","changed_gmt":"2024-07-24 19:50:46","author":"Tatianna Richardson","boilerplate_text":"","field_publication":"","field_article_url":"","field_event_time":{"event_time_start":"2024-08-01T14:00:00-04:00","event_time_end":"2024-08-01T15:00:00-04:00","event_time_end_last":"2024-08-01T15:00:00-04:00","gmt_time_start":"2024-08-01 18:00:00","gmt_time_end":"2024-08-01 19:00:00","gmt_time_end_last":"2024-08-01 19:00:00","rrule":null,"timezone":"America\/New_York"},"location":"ZOOM","extras":[],"groups":[{"id":"221981","name":"Graduate Studies"}],"categories":[],"keywords":[{"id":"102851","name":"Phd proposal"}],"core_research_areas":[],"news_room_topics":[],"event_categories":[{"id":"1788","name":"Other\/Miscellaneous"}],"invited_audience":[{"id":"78771","name":"Public"}],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[],"email":[],"slides":[],"orientation":[],"userdata":""}}}