{"686893":{"#nid":"686893","#data":{"type":"event","title":"PhD Defense by  Ali Hassani","body":[{"value":"\u003Cp\u003E\u003Cstrong\u003ETitle:\u003C\/strong\u003E\u0026nbsp;Neighborhood Attention: Fast and Flexible Sparse Attention\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EAli Hassani\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EPh.D. Student in Computer Science\u003C\/p\u003E\u003Cp\u003ESchool of Interactive Computing\u003C\/p\u003E\u003Cp\u003EGeorgia Institute of Technology\u003C\/p\u003E\u003Cp\u003E\u003Ca href=\u0022https:\/\/alihassanijr.com\u0022 target=\u0022_blank\u0022 title=\u0022https:\/\/alihassanijr.com\u0022\u003Ealihassanijr.com\u003C\/a\u003E\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EDate:\u003C\/strong\u003E\u0026nbsp;Wednesday, January 7th, 2026\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003ETime:\u003C\/strong\u003E\u0026nbsp;13:00-15:00 EST\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003ELocation:\u003C\/strong\u003E\u0026nbsp;Coda C1115 Druid Hills\u003C\/p\u003E\u003Cp\u003ERemote option (Zoom):\u003C\/p\u003E\u003Cp\u003E\u2002\u2002\u2002\u2002\u003Ca href=\u0022https:\/\/gatech.zoom.us\/j\/92667338016\u0022\u003Ehttps:\/\/gatech.zoom.us\/j\/92667338016\u003C\/a\u003E\u003C\/p\u003E\u003Cp\u003E\u2002\u2002\u2002\u2002Meeting ID: 926 6733 8016\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003ECommittee:\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EDr. \u003Ca href=\u0022mailto:shi@gatech.edu\u0022 id=\u0022OWAAM258731\u0022\u003E@Shi, Humphrey\u003C\/a\u003E\u0026nbsp;(Advisor)\u0026nbsp;- School of Interactive Computing, Georgia Institute of Technology\u003C\/p\u003E\u003Cp\u003EDr. \u003Ca href=\u0022mailto:w-hwu@illinois.edu\u0022 id=\u0022OWAAM518120\u0022\u003E@Hwu, Wen-mei\u003C\/a\u003E\u0026nbsp;- Electrical \u0026amp; Computer Engineering, University of Illinois at Urbana-Champaign\u003C\/p\u003E\u003Cp\u003EDr. \u003Ca href=\u0022mailto:kartikgo@gatech.edu\u0022 id=\u0022OWAAM212107\u0022\u003E@Goyal, Kartik\u003C\/a\u003E\u0026nbsp;- School of Interactive Computing, Georgia Institute of Technology\u003C\/p\u003E\u003Cp\u003EDr. \u003Ca href=\u0022mailto:judy@gatech.edu\u0022 id=\u0022OWAAM846771\u0022\u003E@Hoffman, Judy\u003C\/a\u003E\u0026nbsp;- School of Interactive Computing, Georgia Institute of Technology\u003C\/p\u003E\u003Cp\u003EDr. \u003Ca href=\u0022mailto:zkira@gatech.edu\u0022 id=\u0022OWAAM641284\u0022\u003E@Kira, Zsolt\u003C\/a\u003E\u0026nbsp;- School of Interactive Computing, Georgia Institute of Technology\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EAbstract:\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EAttention is at the heart of most foundational AI models, across tasks and modalities.\u003C\/p\u003E\u003Cp\u003EIn many of those cases, it incurs a significant amount of computation, which is quadratic\u003C\/p\u003E\u003Cp\u003Ein complexity, and often cited as one of its greatest limitations. As a result, many sparse\u003C\/p\u003E\u003Cp\u003Eapproaches have been proposed to alleviate this issue, with one of the most common\u003C\/p\u003E\u003Cp\u003Eapproaches being masked or reduced attention span.\u003C\/p\u003E\u003Cp\u003EIn this work, we revisit sliding window approaches, which were commonly believed to\u003C\/p\u003E\u003Cp\u003Ebe inherently inefficient, and we propose a new framework called Neighborhood Attention\u003C\/p\u003E\u003Cp\u003E(NA). Through it, we solve design flaws in the original sliding window attention works, at-\u003C\/p\u003E\u003Cp\u003Etempt to implement the approach efficiently for modern hardware accelerators, specifically\u003C\/p\u003E\u003Cp\u003EGPUs, and conduct experiments that highlight the strengths and weaknesses of these\u0026nbsp;\u003C\/p\u003E\u003Cp\u003Eapproaches. At the same time, we bridge the parameterization and properties of\u003C\/p\u003E\u003Cp\u003EConvolution and Attention, by showing that NA exhibits inductive biases and receptive fields\u003C\/p\u003E\u003Cp\u003Esimilar to that in convolutions, while still capable of capturing inter-dependencies, both short\u003C\/p\u003E\u003Cp\u003Eand long range, similar to attention.\u003C\/p\u003E\u003Cp\u003EWe then show the necessity for and challenges that arise from infrastructure, especially\u003C\/p\u003E\u003Cp\u003Ein the context of modern implementations such as Flash Attention, and develop even more\u003C\/p\u003E\u003Cp\u003Eefficient and performance-optimized implementations for NA, specifically for the most re-\u003C\/p\u003E\u003Cp\u003Ecent and popular AI hardware accelerators, the NVIDIA Hopper and Blackwell GPUs.\u003C\/p\u003E\u003Cp\u003EWe build models based on the NA family, highlighting its superior quality and efficiency\u003C\/p\u003E\u003Cp\u003Ecompared to existing approaches, and also plug NA into existing foundational models,\u003C\/p\u003E\u003Cp\u003Eand showing that it can accelerate those models by up to 1.6\u00d7 end-to-end and without\u003C\/p\u003E\u003Cp\u003Efurther training, and up to 2.6\u00d7 end-to-end with training. We further demonstrate that our\u003C\/p\u003E\u003Cp\u003Emethodology can actually create sparse Attention patterns that realize the theoretical limit\u003C\/p\u003E\u003Cp\u003Eof their speedups.\u003C\/p\u003E\u003Cp\u003EThis work is open-sourced through the NATTEN project at natten.org.\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EThesis PDF:\u003C\/strong\u003E \u003Ca href=\u0022https:\/\/alihassanijr.com\/files\/Hassani-Dissertation-2025-10-11.pdf\u0022\u003Ehttps:\/\/alihassanijr.com\/files\/Hassani-Dissertation-2025-10-11.pdf\u003C\/a\u003E\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E","summary":"","format":"limited_html"}],"field_subtitle":"","field_summary":[{"value":"\u003Cp\u003ENeighborhood Attention: Fast and Flexible Sparse Attention\u003C\/p\u003E","format":"limited_html"}],"field_summary_sentence":[{"value":"Neighborhood Attention: Fast and Flexible Sparse Attention"}],"uid":"27707","created_gmt":"2025-12-15 17:37:48","changed_gmt":"2025-12-15 17:38:30","author":"Tatianna Richardson","boilerplate_text":"","field_publication":"","field_article_url":"","field_event_time":{"event_time_start":"2026-01-07T13:00:00-05:00","event_time_end":"2026-01-07T15:00:00-05:00","event_time_end_last":"2026-01-07T15:00:00-05:00","gmt_time_start":"2026-01-07 18:00:00","gmt_time_end":"2026-01-07 20:00:00","gmt_time_end_last":"2026-01-07 20:00:00","rrule":null,"timezone":"America\/New_York"},"location":"Coda C1115 Druid Hills","extras":[],"groups":[{"id":"221981","name":"Graduate Studies"}],"categories":[],"keywords":[{"id":"100811","name":"Phd Defense"}],"core_research_areas":[],"news_room_topics":[],"event_categories":[{"id":"1788","name":"Other\/Miscellaneous"}],"invited_audience":[{"id":"78771","name":"Public"}],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[],"email":[],"slides":[],"orientation":[],"userdata":""}}}