{"645182":{"#nid":"645182","#data":{"type":"event","title":"PhD Defense by Jun-Kun Wang","body":[{"value":"\u003Cp\u003E\u003Cstrong\u003ETitle:\u003C\/strong\u003E Understanding Modern Techniques in Optimization: Frank-Wolfe, Nesterov\u0026#39;s Momentum, and Polyak\u0026#39;s Momentum\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003EJun-Kun Wang\u003C\/p\u003E\r\n\r\n\u003Cp\u003EPh.D. Candidate\u003C\/p\u003E\r\n\r\n\u003Cp\u003ESchool of Computer Science, Georgia Institute of Technology\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cstrong\u003EDate:\u003C\/strong\u003E March 23th, 2021 (Tuesday)\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cstrong\u003ETime:\u003C\/strong\u003E 1:00 PM - 2:30 PM (EST)\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cstrong\u003ELocation:\u003C\/strong\u003E\u0026nbsp;*No Physical Location*\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cstrong\u003EBlueJeans:\u003C\/strong\u003E\u0026nbsp;\u003Ca href=\u0022https:\/\/nam12.safelinks.protection.outlook.com\/?url=https%3A%2F%2Fbluejeans.com%2F601294912\u0026amp;data=04%7C01%7Ctatianna.richardson%40grad.gatech.edu%7Cc892c5b8a74f446f11c208d8e3d01a76%7C482198bbae7b4b258b7a6d7f32faa083%7C0%7C0%7C637509830224763137%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000\u0026amp;sdata=oLciAYYXX5kZ621%2FtcRBudCffZlM9QlTBOZGI5LSmGE%3D\u0026amp;reserved=0\u0022 id=\u0022LPlnk230460\u0022\u003Ehttps:\/\/bluejeans.com\/601294912\u003C\/a\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cstrong\u003ECommittee:\u003C\/strong\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003EDr. Jacob Abernethy (advisor) - School of Computer Science, Georgia Institute of Technology\u003C\/p\u003E\r\n\r\n\u003Cp\u003EDr. Guanghui Lan - Industrial \u0026amp; System Engineering, Georgia Institute of Technology\u003C\/p\u003E\r\n\r\n\u003Cp\u003EDr. Vidya Muthukumar - Industrial \u0026amp; System Engineering and School of Electrical and Computer Engineering, Georgia Institute of Technology\u003C\/p\u003E\r\n\r\n\u003Cp\u003EDr. Richard Peng - School of Computer Science, Georgia Institute of Technology\u003C\/p\u003E\r\n\r\n\u003Cp\u003EDr. Santosh Vempala - School of Computer Science, Georgia Institute of Technology\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cstrong\u003EAbstract:\u003C\/strong\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003EOptimization is essential in machine learning, statistics, and data science. Among the first-order optimization algorithms, the popular ones include the Frank-Wolfe method, Nesterov\u0026#39;s accelerated methods, and Polyak\u0026#39;s momentum. While theoretical analysis of the Frank-Wolfe method and the Nesterov\u0026#39;s methods are available in the literature, the analysis can be quite complicated or less intuitive. Polyak\u0026#39;s momentum, on the other hand, is widely used in training neural networks and is currently the default choice of momentum in Pytorch and Tensorflow. It is widely observed that Polyak\u0026#39;s momentum helps to train a neural network faster, compared with the case without momentum. However, there are very few examples that exhibit a provable acceleration via Polyak\u0026#39;s momentum, compared to vanilla gradient descent. There is an apparent gap between the theory and the practice of Polyak\u0026#39;s momentum.\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003EIn the first part of this dissertation research, we develop a modular framework that can serve as a recipe for constructing and analyzing iterative algorithms for convex optimization. Specifically, our work casts optimization as iteratively playing a two-player zero-sum game. Many existing optimization algorithms including Frank-Wolfe and Nesterov\u0026#39;s acceleration methods can be recovered from the game by pitting two online learners with appropriate strategies against each other. Furthermore, the sum of the weighted average regrets of the players in the game implies the convergence rate. As a result, our approach provides simple alternative proofs to these algorithms. Moreover, we demonstrate that our approach of ``optimization as iteratively playing a game\u0026#39;\u0026#39; leads to three new fast Frank-Wolfe-like algorithms for some constraint sets, which further shows that our framework is indeed generic, modular, and easy-to-use.\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003EIn the second part, we develop a modular analysis of provable acceleration via Polyak\u0026#39;s momentum for certain problems, which include solving the classical strongly quadratic convex problems, training a wide ReLU network under the neural tangent kernel regime, and training a deep linear network with an orthogonal initialization. We develop a meta theorem and show that when applying Polyak\u0026rsquo;s momentum for these problems, the induced dynamics exhibit a form where we can directly apply our meta theorem.\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003EIn the last part of the dissertation, we show another advantage of the use of Polyak\u0026#39;s momentum --- it facilitates fast saddle point escape in smooth non-convex optimization. This result, together with those of the second part, sheds new light on Polyak\u0026#39;s momentum in modern non-convex optimization and deep learning.\u003C\/p\u003E\r\n","summary":null,"format":"limited_html"}],"field_subtitle":"","field_summary":"","field_summary_sentence":[{"value":"Understanding Modern Techniques in Optimization: Frank-Wolfe, Nesterov\u0027s Momentum, and Polyak\u0027s Momentum"}],"uid":"27707","created_gmt":"2021-03-10 14:41:31","changed_gmt":"2021-03-10 14:41:31","author":"Tatianna Richardson","boilerplate_text":"","field_publication":"","field_article_url":"","field_event_time":{"event_time_start":"2021-03-23T14:00:00-04:00","event_time_end":"2021-03-23T15:30:00-04:00","event_time_end_last":"2021-03-23T15:30:00-04:00","gmt_time_start":"2021-03-23 18:00:00","gmt_time_end":"2021-03-23 19:30:00","gmt_time_end_last":"2021-03-23 19:30:00","rrule":null,"timezone":"America\/New_York"},"extras":[],"groups":[{"id":"221981","name":"Graduate Studies"}],"categories":[],"keywords":[{"id":"100811","name":"Phd Defense"}],"core_research_areas":[],"news_room_topics":[],"event_categories":[{"id":"1788","name":"Other\/Miscellaneous"}],"invited_audience":[{"id":"78761","name":"Faculty\/Staff"},{"id":"78771","name":"Public"},{"id":"174045","name":"Graduate students"},{"id":"78751","name":"Undergraduate students"}],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[],"email":[],"slides":[],"orientation":[],"userdata":""}}}