{"673072":{"#nid":"673072","#data":{"type":"event","title":"PhD Defense by Mikhail Isaev","body":[{"value":"\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cstrong\u003E\u003Cspan\u003ETitle:\u0026nbsp;Methodologies for co-designing supercomputer-scale systems and deep learning software\u003C\/span\u003E\u003C\/strong\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cstrong\u003E\u003Cspan\u003EMikhail Isaev\u003C\/span\u003E\u003C\/strong\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003EComputer Science PhD Student\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003ESchool of Computational Science and Engineering\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003EGeorgia Institute of Technology\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cstrong\u003E\u003Cspan\u003EDate: \u003C\/span\u003E\u003C\/strong\u003E\u003Cspan\u003ETuesday, Feb 20, 2024\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cstrong\u003E\u003Cspan\u003ETime:\u003C\/span\u003E\u003C\/strong\u003E\u003Cspan\u003E\u0026nbsp;13:00 \u2013 15:00 EST\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cstrong\u003E\u003Cspan\u003ELocation:\u003C\/span\u003E\u003C\/strong\u003E\u003Cspan\u003E\u0026nbsp;C1015 (Vinings) and \u003Ca href=\u0022https:\/\/teams.microsoft.com\/l\/meetup-join\/19%3ameeting_YTAxZWVjMGQtYmU3My00Y2U3LWI2ODUtNzZiY2U1MjMwODQ2%40thread.v2\/0?context=%7b%22Tid%22%3a%22482198bb-ae7b-4b25-8b7a-6d7f32faa083%22%2c%22Oid%22%3a%226cbfbeb2-24c1-450e-b11b-3e2ce756bbc2%22%7d\u0022 title=\u0022https:\/\/teams.microsoft.com\/l\/meetup-join\/19%3ameeting_YTAxZWVjMGQtYmU3My00Y2U3LWI2ODUtNzZiY2U1MjMwODQ2%40thread.v2\/0?context=%7b%22Tid%22%3a%22482198bb-ae7b-4b25-8b7a-6d7f32faa083%22%2c%22Oid%22%3a%226cbfbeb2-24c1-450e-b11b-3e2ce756bbc2%22%7d\u0022\u003ETeams\u003C\/a\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Ca href=\u0022https:\/\/teams.microsoft.com\/l\/meetup-join\/19%3ameeting_YTAxZWVjMGQtYmU3My00Y2U3LWI2ODUtNzZiY2U1MjMwODQ2%40thread.v2\/0?context=%7b%22Tid%22%3a%22482198bb-ae7b-4b25-8b7a-6d7f32faa083%22%2c%22Oid%22%3a%226cbfbeb2-24c1-450e-b11b-3e2ce756bbc2%22%7d\u0022 target=\u0022_blank\u0022\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cimg alt=\u0022\u0022 src=\u0022data:image\/png;base64,iVBORw0KGgoAAAANSUhEUgAAAGAAAABgCAYAAADimHc4AAAAAXNSR0IArs4c6QAACnVJREFUeAHtXXtsHEcZ\/2Z27fP5dbbPbVNoKyFUBbWFQhLzbJVatNCHUkLch5pQFUVVlMAfVAGhEp8lE9shRUIIIQpKBRWJ46CmiVMQ5I+WtFVatSVNoGlBBDUKSUMdiJ3g9+P2dvhmz3c+Nrd3s7uzc+tmVzrd7M433\/fN77f7zet2DiA6IgQiBCIEIgQuVwTIYq141w8ufgoyxmoTyK0A7MOEwYd4XRiB9wHIvyiww6DpB3q+1\/znMNdx0RGQ2jZyH2RYLwO2VARYAuQEaCTVuyX5rIi8aplFQ0Cq578fYcQYAMY+6wkkQl4nTF\/b29V0ylP5gAotCgK6tl1caZqZfQh+0hcOhIxQqnX0bGl+2ZceiYVDT8A8+M8j+FVS6k1IGkm4IywkhJqA+bBzxPedb2cOnwQMR21hCEfU7luYzudjvr+wU6xCGMos3cXyFF8LLQFWb8drgysCIuq2bIjIBiijB6jbn2rsavpTIFA6a8NT9\/SrXzt2MzGN+0zG7iCEXAcMkkBghDF2hhLyPKP6s4P9y94q50Uo2wA+yDIN41g552XkU11f5mawtvqRNz9K0uYPEeg15ewjMftZFf3ugV+vOOkkG84QhCNcJ4elX3dha\/W6I3eROfOoCPjcTy7H5Xk5J79DSUB2esHJZbnXRW1Z4Jvm73AEnnDjAZcnWM6JhFASwOd23FTSn2x5W1bYMdkeBqB5scXLESzP9djLh5KA3MSa3dkgzkVsWTHf5Z1v99V6ErDtsF8PJQF2J4M8ZwQpKHHw3o5ozC+hxsrieri+QrlQEpCdUi50M8g0GyqlnXc1S+W7zbPrCyUBfD7fbcW8y5e2xfv53nVfWtKuL5QDsYmx9\/\/EGGm\/1H3nK9jnxvWXGMRiDYCTbc6Cthxr4cZ2rfCUD7IwdBRe8pW2Bm0FGsL5BNDYvgIfhZIcJCM9A5OT5wGnroXKWEK4alZSmI9wZR42faEk4Cfblx7BO+VdT\/XGm3V2dlyoKF8tKzsKxukFIWWiQjZ9Vghas+7Fa2ZnsYvE2G2o52pRXUHKvXvyXFH1jKWBmWNgpt\/DUAMQr03ClUtugOrq2rx8xpjNp0smcKmyZD5m4pN1Br+kYTKvL2+WZsHPvIWWHpJpKG9BcoKQKqBaErSamzDUAEyMD8E\/T74Mc3NTeUtCMRuXKEXWifnEWl6xhIRdH83e+dAiQbdSFQR0oFXXWjZN04D\/nPubuP3sgsxakQJ8VlNETlTGrg87AVbYES0fKjlCG\/P+TE8JhurskmSH6GoYn1LG9mh\/3pCPBNdjn6LmjbC0+ObDN09FeTjKHZnMXC7p\/J1dlHe9HsynlLHBHnVWXD6Hl+d67JKh7AXZnZRynv1ZSpuXxXg+n88oeQgXT1z0bxe85uV4+WLrAh94AnhXk2j0\/r5U6+dEw84CdAupA7vbDjJKV7l9Eqw7H8vx8gvaFlKhHAkvuFc+pWnV72D4aUXJJuw08jv0HBA6RAkckv3TRA4iTikvl7kiRu7peEHeOLs8XtIlfr\/v9oosq8paE3b9BOx66hZIJKqlA8kVjo7NwcOPvhKIbtlK53szfNG9y49u1wRQfLb5J4gjKL1B+CpL5we+EZYFVFB6XD8BQTniVe+X7t6Wb8M+duOqvJqGxiX5tJcEn86YmZ2D0dEJuDAyij\/78X9gj2iOEfY2JfQZ7SsrfrT3AZJZ9AT4h6W4Bhy1QrwmZn0a6mvh9Okh3yTgunA1KlluMnO5OXjkrvufYbe7JmBsIo0LVsWdzl1tbKgCXgH7wRvZUsf4OOoO4VFXF4eWZAJG8EmQd7DbMs+9+W3XBGz61utlfej\/5a3QZOsppdMmrFt\/uGzZsAokEvWSCQDAJ+GBqBEWZLwmJr\/rTRj5eESAIAHFQqpgUUcx3iZEBDjCoyYjIkANzo5WIgIcoVGTERGgBmdHKxEBjtCoyYgIUIOzo5WIAEdo1GREBKjB2dFKRIAjNGoyIgLU4OxoJSLAERo1GREBanB2tBIR4AiNmoyIADU4O1qJCHCERk1GRIAanB2tRAQ4QqMmIyJADc6OViICHKFRkxERoAZnRyuuf5biqKkgI5NhkMngG3QFh2H8\/3lB1mWdDISARza8clmDKlx5QoajECSMViCCZyMCAsFVWOl7EQHCWAUhSI5GBASBq6DOKgq\/4QSU3LBIUFdFxMzMTEXsyjCKP3U8tre\/7QTF35G\/JENhJXSk08PKzDK+MYXMg5CfcXU0FsO3twlckKlbhS6TzcHU+DsqTFk2DNu4xo9hvPtf3d+\/4mmug+7f3X42FtNuxidhD56HPhzxsDM7cxZGhw\/htjUL29K42SXLC3iG4ekl+UtMIfhp3MRyI35bbz1ZAzFOAkoK7R5yiUYXF25Z+Z2\/Gsb0DS6KlBTVq+L5fF2vyaeDSGSkEICgE7JxcFdb\/tFV2guimn5CGjj4ClRNTXa3FEKotXGTNN1FFM3M+n19ah783W2\/KlSvlABdjx8tNO4lzcHmd359\/ZWg4yZ91dV10Ji4BjRtYVYFH28vqkuWmZqaLplfKhO9mcBXq9cfGGjbYZdb8NqeE8B5csmyXVexqa34ZkigxOP+EXK9Z8ScnJrx5DPeDAd1nW7cu3PFmWJOeVJaTJHItb07V50hmvYXEVmvMvzu51tXyjxQ5RvYDf089hb\/gPrLxiL+PjB+BvFhXTU48Om7ncDnPip9ArhBDCHP4dcynpZ5ENwrVK+qgVh1Pe4pJ7ta5OnBPZ95Df29Z+3a483TdPpeZpJPIjH8bXDc8IrFsHU9jeenMP33OKv57cDAJy6K1E9+sCxjdcOGF1szmnkK30SvLyPqKjte2wxNzde5KiMkjLtsJWuT127eTLw3AiUMKQ1B3I8dO9qHqaY9VcIn11l8DFDfcJXrciIFsO+yIyjwuX3lBHCjuGPCNoylEzzt5+DA18QTkLzieqtH5EdX0bKEjMdp\/KdF8yRdVB6Ccn6neoY3YW\/oydx5GL+xIf1Gb1frz4P0rWIE8Eqles8fxE1J7gyygl51IzCHe1KtK\/FJlbFRiqMbFQlBOW\/ipHY9dosEN\/zMlVLwjaFH12KPBg0+r0lFCejsrBvSCH0YSSjbt1YAu2XC6sMzWNO9pfEfKmxWlABewa2dLQdxt\/8HseKGigqXtIHhhhH6dYz7L5SUk5hZcQJ4XbamWnHUSNbhkyBnztcLQGgb4d\/Ul2rh0\/LKjoo2wvZapnpG7sRpl524q9QV9rwgzxEEXFojD+KdfyhIO8V0h4oA7mBf3+TVU2x6N24q3l7MYenXCBzDGZk1fanm09J1CygMHQHc527GaLrvwmM4r7IFiZD7FyLzoGDFh7GX833tC8lfdLdXrv0JJQG5G+eJJ843jBv0MdzbazMO2nBrYv8HVngKJwSfrNNJ7+OPt8jcBM6Tc6EmIFej7h9fbDImzA5sHzpwm68v8p2mcnlC37yBBfgjjqj6q5qTg93f9D8NImRXQGhREFBYj+3bLyQmM+zLOD9\/E15fyj\/4b3jz06DMwHFrBv8Ibgwb1bfxc5xQdhynh1\/r7Kz\/d6GeKB0hECEQIRAhAP8Df5YlCtXXo4oAAAAASUVORK5CYII=\u0022 \/\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/a\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Ca href=\u0022https:\/\/teams.microsoft.com\/l\/meetup-join\/19%3ameeting_YTAxZWVjMGQtYmU3My00Y2U3LWI2ODUtNzZiY2U1MjMwODQ2%40thread.v2\/0?context=%7b%22Tid%22%3a%22482198bb-ae7b-4b25-8b7a-6d7f32faa083%22%2c%22Oid%22%3a%226cbfbeb2-24c1-450e-b11b-3e2ce756bbc2%22%7d\u0022 target=\u0022_blank\u0022\u003EJoin conversation\u003C\/a\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003Eteams.microsoft.com\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cstrong\u003E\u003Cspan\u003ECommittee:\u003C\/span\u003E\u003C\/strong\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003EDr. Richard W. Vuduc (advisor), School of Computational Science and Engineering, Georgia Institute of Technology\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003EDr. Jeffrey Young, School of Computer Science, Georgia Institute of Technology\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003EDr. Tushar Krishna, School of Electrical and Computer Engineering, Georgia Institute of Technology\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003EDr. Hyesoon Kim, School of Computer Science, Georgia Institute of Technology\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003EDr. Nicholas G. McDonald, Nvidia Research\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cstrong\u003E\u003Cspan\u003EAbstract:\u003C\/span\u003E\u003C\/strong\u003E\u0026nbsp;\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003EThis dissertation introduces new methodologies to co-design deep learning software and supercomputer hardware for large-scale training.\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003EThe first is an analytical performance model to co-design large language models (LLM) and supercomputer architectures during the early phases of the system design process. On the algorithm side, we consider diverse implementation strategies, including data, tensor, and pipeline parallelism, communication-computation overlap, and memory optimization. The hardware aspect includes hierarchical memory systems, multiple interconnection networks, and parameterized efficiencies based on operation size. We implement it in Calculon, an open-source tool that allows estimating performance for billions of strategy and architecture combinations. This facilitates co-design-space exploration for future LLMs with trillions of parameters, yielding insights into optimal system characteristics and the interplay between algorithmic and architectural decisions.\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003EAs models exceed 100 trillion parameters, memory capacity and network speed become critical bottlenecks. For the former, Calculon suggests adding slower, high-capacity memory to store all intermediate tensors and model parameters while utilizing faster memory solely for current computation. For the latter, we present novel distributed-memory parallel matrix multiplication algorithms capable of hiding communication entirely, potentially achieving perfect scaling.\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003ELooking ahead, we foresee a need to model artificial intelligence (AI) applications beyond LLMs and perform detailed system simulations in later design stages. Our second open-source tool, ParaGraph, translates compiled parallel programs into high-level graphs for emulator-based dynamic execution in network simulation environments. Case studies on deep learning workloads extracted from JAX and TensorFlow programs illustrate ParaGraph\u0027s utility for software-hardware co-design workflows, including communication optimization and hardware bottleneck identification.\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/p\u003E\r\n","summary":"","format":"limited_html"}],"field_subtitle":"","field_summary":[{"value":"\u003Cp\u003E\u003Cstrong\u003E\u003Cspan\u003E\u003Cspan\u003E\u003Cspan\u003EMethodologies for co-designing supercomputer-scale systems and deep learning software\u003C\/span\u003E\u003C\/span\u003E\u003C\/span\u003E\u003C\/strong\u003E\u003C\/p\u003E\r\n","format":"limited_html"}],"field_summary_sentence":[{"value":"Methodologies for co-designing supercomputer-scale systems and deep learning software"}],"uid":"27707","created_gmt":"2024-02-20 16:00:20","changed_gmt":"2024-02-20 16:00:20","author":"Tatianna Richardson","boilerplate_text":"","field_publication":"","field_article_url":"","field_event_time":{"event_time_start":"2024-02-20T13:00:00-05:00","event_time_end":"2024-02-20T15:00:00-05:00","event_time_end_last":"2024-02-20T15:00:00-05:00","gmt_time_start":"2024-02-20 18:00:00","gmt_time_end":"2024-02-20 20:00:00","gmt_time_end_last":"2024-02-20 20:00:00","rrule":null,"timezone":"America\/New_York"},"location":"C1015 (Vinings) and Teams","extras":[],"groups":[{"id":"221981","name":"Graduate Studies"}],"categories":[],"keywords":[{"id":"100811","name":"Phd Defense"}],"core_research_areas":[],"news_room_topics":[],"event_categories":[{"id":"1788","name":"Other\/Miscellaneous"}],"invited_audience":[{"id":"78771","name":"Public"}],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[],"email":[],"slides":[],"orientation":[],"userdata":""}}}