{"676869":{"#nid":"676869","#data":{"type":"event","title":"ML@GT Seminar Series | A Picture of the Prediction Space of Deep Neural Networks","body":[{"value":"\u003Cp\u003EFeaturing Pratik Chaudhari, University of Pennsylvania\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EAbstract:\u003C\/strong\u003E\u0026nbsp;I will argue that deep networks work well because of a characteristic structure in the space of learnable tasks. The input correlation matrix for typical tasks has a \u201csloppy\u201d eigenspectrum where eigenvalues decay linearly on a logarithmic scale. As a consequence, the Hessian and the Fisher Information Matrix of a trained network also have a sloppy eigenspectrum. Using this idea, I will demonstrate an analytical, non-vacuous PAC-Bayes bound on the generalization error for general deep networks.\u003C\/p\u003E\u003Cp\u003EI will show that the training process in deep learning explores a remarkably low dimensional manifold, as low as three. Networks with a wide variety of architectures, sizes, optimization and regularization methods lie on the same manifold. Networks being trained on different tasks (e.g., different subsets of ImageNet) using different methods (e.g., supervised, transfer, meta, semi and self-supervised learning) also lie on the same low-dimensional manifold.\u003C\/p\u003E\u003Cp\u003EI will show that typical tasks are highly redundant functions of their inputs. Many perception tasks, from visual recognition, semantic segmentation, optical flow, depth estimation, to vocalization discrimination, can be predicted extremely well regardless whether data is projected in the principal subspace where it varies the most, some intermediate subspace with moderate variability---or the bottom subspace where data varies the least.\u003C\/p\u003E\u003Cp\u003EReferences:\u0026nbsp;\u003C\/p\u003E\u003Col\u003E\u003Cli\u003EDoes the data induce capacity control in deep learning? Rubing Yang, Jialin Mao, and Pratik Chaudhari. [ICML \u201922] \u003Ca href=\u0022https:\/\/arxiv.org\/abs\/2110.14163\u0022\u003Ehttps:\/\/arxiv.org\/abs\/2110.14163\u003C\/a\u003E\u003C\/li\u003E\u003Cli\u003EThe Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold. Jialin Mao, Itay Griniasty, Han Kheng Teoh, Rahul Ramesh, Rubing Yang, Mark K. Transtrum, James P. Sethna, Pratik Chaudhari. [PNAS 2024]. \u003Ca href=\u0022https:\/\/arxiv.org\/abs\/2305.01604\u0022\u003Ehttps:\/\/arxiv.org\/abs\/2305.01604\u003C\/a\u003E\u003C\/li\u003E\u003Cli\u003EA picture of the space of typical learnable tasks. Rahul Ramesh, Jialin Mao, Itay Griniasty, Rubing Yang, Han Kheng Teoh, Mark Transtrum, James P. Sethna, and Pratik Chaudhari [ICML \u201923]. \u003Ca href=\u0022https:\/\/arxiv.org\/abs\/2210.17011\u0022\u003Ehttps:\/\/arxiv.org\/abs\/2210.17011\u003C\/a\u003E\u003C\/li\u003E\u003Cli\u003EMany Perception Tasks are Highly Redundant Functions of their Input Data. Rahul Ramesh, Anthony Bisulco, Ronald W. DiTullio, Linran Wei, Vijay Balasubramanian, Kostas Daniilidis, Pratik Chaudhari. \u003Ca href=\u0022https:\/\/arxiv.org\/abs\/2407.13841\u0022\u003Ehttps:\/\/arxiv.org\/abs\/2407.13841\u003C\/a\u003E\u003C\/li\u003E\u003C\/ol\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EBio:\u0026nbsp;\u003C\/strong\u003EPratik Chaudhari is an Assistant Professor in Electrical and Systems Engineering and Computer and Information Science at the University of Pennsylvania. He is a core member of the GRASP Laboratory. From 2018-19, he was a Senior Applied Scientist at Amazon Web Services and a Postdoctoral Scholar in Computing and Mathematical Sciences at Caltech. Pratik received his PhD in Computer Science from UCLA, and his Master\u0027s and Engineer\u0027s degrees in Aeronautics and Astronautics from MIT. He was a part of NuTonomy Inc. (now Hyundai-Aptiv Motional) from 2014-16. He is the recipient of the Amazon Machine Learning Research Award, NSF CAREER award and the Intel Rising Star Faculty Award.\u003C\/p\u003E","summary":"","format":"limited_html"}],"field_subtitle":"","field_summary":[{"value":"\u003Cp\u003EMachine Learning Center Seminar Series is held bi-weekly on Wednesdays at 12pm.\u0026nbsp;\u003C\/p\u003E","format":"limited_html"}],"field_summary_sentence":[{"value":"Featuring Pratik Chaudhari, University of Pennsylvania "}],"uid":"36518","created_gmt":"2024-09-17 14:56:42","changed_gmt":"2024-10-10 15:14:49","author":"shatcher8","boilerplate_text":"","field_publication":"","field_article_url":"","field_event_time":{"event_time_start":"2024-10-23T12:00:00-04:00","event_time_end":"2024-10-23T13:00:00-04:00","event_time_end_last":"2024-10-23T13:00:00-04:00","gmt_time_start":"2024-10-23 16:00:00","gmt_time_end":"2024-10-23 17:00:00","gmt_time_end_last":"2024-10-23 17:00:00","rrule":null,"timezone":"America\/New_York"},"location":"CODA 9th Floor Atrium","extras":["free_food"],"hg_media":{"675273":{"id":"675273","type":"image","title":"2024.1023 ML Seminar Announcement-Pratik Chaudhari.jpg","body":null,"created":"1728573173","gmt_created":"2024-10-10 15:12:53","changed":"1728573173","gmt_changed":"2024-10-10 15:12:53","alt":"ML@GT Seminar Series welcomes guest Pratik Chaudhari on Wednesday, October 23 at 12pm","file":{"fid":"258877","name":"2024.1023 ML Seminar Announcement-Pratik Chaudhari.jpg","image_path":"\/sites\/default\/files\/2024\/10\/10\/2024.1023%20ML%20Seminar%20Announcement-Pratik%20Chaudhari.jpg","image_full_path":"http:\/\/hg.gatech.edu\/\/sites\/default\/files\/2024\/10\/10\/2024.1023%20ML%20Seminar%20Announcement-Pratik%20Chaudhari.jpg","mime":"image\/jpeg","size":163944,"path_740":"http:\/\/hg.gatech.edu\/sites\/default\/files\/styles\/740xx_scale\/public\/2024\/10\/10\/2024.1023%20ML%20Seminar%20Announcement-Pratik%20Chaudhari.jpg?itok=WMFjV5ZL"}}},"media_ids":["675273"],"related_links":[{"url":"https:\/\/ml.gatech.edu\/","title":""}],"groups":[{"id":"576481","name":"ML@GT"}],"categories":[],"keywords":[{"id":"173555","name":"Center for Machine Learning"},{"id":"9167","name":"machine learning"},{"id":"178072","name":"Deep Neural Networks"}],"core_research_areas":[],"news_room_topics":[],"event_categories":[{"id":"1795","name":"Seminar\/Lecture\/Colloquium"}],"invited_audience":[{"id":"78761","name":"Faculty\/Staff"},{"id":"177814","name":"Postdoc"},{"id":"174045","name":"Graduate students"},{"id":"78751","name":"Undergraduate students"}],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[{"value":"\u003Cp\u003EShelli Hatcher, Program and Operations Manager\u003C\/p\u003E","format":"limited_html"}],"email":[],"slides":[],"orientation":[],"userdata":""}}}