A wave of recent work has sought to understand how pretrained language models work. Such analyses have resulted in two seemingly contradictory sets of results. On one hand, work based on "probing classifiers" generally suggests that SOTA language models contain rich information about linguistic structure (e.g., parts of speech, syntax, semantic roles). On the other hand, work which measures performance on linguistic "challenge sets" shows that models consistently fail to use this information when making predictions. In this talk, I will present a series of results that attempt to bridge this gap. Our recent experiments suggest that the disconnect is not due to catastrophic forgetting nor is it (entirely) explained by insufficient training data. Rather, it is best explained in terms of how "accessible" features are to the model following pretraining, where "accessibility" can be quantified using an information-theoretic interpretation of probing classifiers.
Ellie Pavlick is an Assistant Professor of Computer Science at Brown University where she leads the Language Understanding and Representation (LUNAR) Lab. She received her PhD from the one-and-only University of Pennsylvania. Her current work focuses on building more cognitively-plausible models of natural language semantics, focusing on grounded language learning and on sample efficiency and generalization of neural language models.
]]>allie.mcfadden@cc.gatech.edu
]]>
Markov decision processes (MDPs) is a minimalist framework to capture that many tasks require long-term plans and feedback due to noisy dynamics. Yet, as a result MDPs lack structure and as such planning and learning in MDPs with the typically enormous state and action spaces is strongly intractable; no algorithm can avoid Bellman's curse of dimensionality in the worst case. However, as recognized already by Bellman and his co-workers at the advent of our field, for many problem of practical interest, the optimal value function of an MDP is well approximated by just using a few basis functions, such as those that are standardly used in numerical calculations. As knowing the optimal value function is essentially equivalent to knowing how to act optimally, one hopes that this observation can be turned into efficient algorithms as there are only a few coefficients to compute. If this is possible, we can think of the resulting algorithms as performing computations with a compressed form of the value functions. While many algorithms have been proposed as early as in the 1960s, until recently not much has been known about whether these compressed computations are possible and when. In this talk, I will discuss a few recent results (some positive, some negative) that are concerned with these compressed computations and conclude with some open problems. As we shall see, still today, there are more open questions than questions that have been satisfactorily answered.
Csaba Szepesvari is a Canada CIFAR AI Chair, the team-lead for the “Foundations” team at DeepMind and a Professor of Computing Science at the University of Alberta. He earned his PhD in 1999 from Jozsef Attila University, in Szeged, Hungary. He has authored three books and over 200 peer-reviewed journal and conference papers. He serves as the action editor of the Journal of Machine Learning Research and Machine Learning, as well as on various program committees. Dr. Szepesvari's interest is artificial intelligence (AI) and, in particular, principled approaches to AI that use machine learning. He is the co-inventor of UCT, a widely successful Monte-Carlo tree search algorithm. UCT ignited much work in AI, such as DeepMind's AlphaGo which defeated the top Go professional Lee Sedol in a landmark game. This work on UCT won the 2016 test-of-time award at ECML/PKDD.
]]>allie.mcfadden@cc.gatech.edu
]]>Deep learning has changed the computing paradigm. Today, AI researchers & practitioners increasingly use deep neural networks for many applications across different modalities and areas such as NLP, Vision, Speech, Conversational and Multimodal AI. However, much of the Deep Learning revolution has been limited to the Cloud and highly specialized hardware. Recently the AI community has witnessed an increasing trend for training larger and larger neural models (e.g., GPT-3, T5, BERT) that achieve state-of-the-art results but require enormous computation, memory and energy resources on the Cloud. In order to enable AI experiences in real-time across all users and devices, ML models have to run efficiently on the Cloud and personal devices on the Edge (e.g., mobile phones, wearables, IoT) which have limited computing capabilities.
In this talk, I will introduce our work on Neural Projection computing, an efficient AI paradigm, and a family of efficient Projection Neural Network architectures that yield fast (e.g., quadratic speedup for transformer networks) and tiny models that shrink memory requirements by upto 10000x while achieving near state-of-the-art performance powering vision and NLP applications on billions of mobile devices. Widespread increase in availability of connected “smart” appliances (e.g., conversational assistants) means that there is an ever-expanding surface area for mobile intelligence and ambient devices in homes. Our approach enables efficient ML to solve complex prediction tasks for such applications both on-device and on Cloud, keeping model size, compute and power usage low while simultaneously optimizing for accuracy.
Dr. Sujith Ravi is a Director at Amazon Alexa AI where he is leading efforts to build the future of multimodal conversational AI experiences at scale. Prior to that, he was leading and managing multiple ML and NLP teams and efforts in Google AI. He founded and headed Google’s large-scale graph-based semi-supervised learning platform, deep learning platform for structured and unstructured data as well as on-device machine learning efforts for products used by billions of people in Search, Ads, Assistant, Gmail, Photos, Android, Cloud and YouTube. These technologies power conversational AI (e.g., Smart Reply), Web and Image Search; On-Device predictions in Android and Assistant; and ML platforms like Neural Structured Learning in TensorFlow, Learn2Compress as Google Cloud service, TensorFlow Lite for edge devices.
Dr. Ravi has authored over 100 scientific publications and patents in top-tier machine learning and natural language processing conferences. His work has been featured in press: Wired, Forbes, Forrester, New York Times, TechCrunch, VentureBeat, Engadget, New Scientist, among others, and also won the SIGDIAL Best Paper Award in 2019 and ACM SIGKDD Best Research Paper Award in 2014. For multiple years, he was a mentor for Google Launchpad startups. Dr. Ravi was the Co-Chair (AI and deep learning) for the 2019 National Academy of Engineering (NAE) Frontiers of Engineering symposium. He was also the Co-Chair for EMNLP 2020, ICML 2019, NAACL 2019, and NeurIPS 2018 ML workshops and regularly serves as Senior/Area Chair and PC of top-tier machine learning and natural language processing conferences like NeurIPS, ICML, ACL, NAACL, AAAI, EMNLP, COLING, KDD, and WSDM.
Website: www.sravi.org
Twitter: @ravisujith
LinkedIn: https://www.linkedin.com/in/sujithravi
]]>allie.mcfadden@cc.gatech.edu
]]>TALK TITLE
Learning Tree Models in Noise: Exact Asymptotics and Robust Algorithms
ABSTRACT
We consider the classical problem of learning tree-structured graphical models but with the twist that the observations are corrupted in independent noise. For the case in which the noise is identically distributed, we derive the exact asymptotics via the use of probabilistic tools from the theory of strong large deviations. Our results strictly improve those of Bresler and Karzand (2020) and Nikolakakis et al. (2019) and demonstrate keen agreement with experimental results for sample sizes as small as that in the hundreds. When the noise is non-identically distributed, Katiyar et al. (2020) showed that although the exact tree structure cannot be recovered, one can recover a "partial" tree structure; that is, one that belongs to the equivalence class containing the true tree. We propose Symmetrized Geometric Averaging (SGA), a statistically robust algorithm for partial tree recovery. We provide error exponent analyses and extensive numerical results on a variety of trees to show that the sample complexity of SGA is significantly better than the algorithm of Katiyar et al. (2020). SGA can be readily extended to Gaussian models and is shown via numerical experiments to be similarly superior.
https://arxiv.org/abs/2101.08917
https://arxiv.org/abs/2005.04354
This is joint work with Anshoo Tandon, Aldric J. Y. Han and Shiyao Zhu.
ABOUT VINCENT
Vincent Y. F. Tan received the B.A. and M.Eng. degrees in electrical and information sciences from Cambridge University and the Ph.D. degree in electrical engineering and computer science (EECS) from the Massachusetts Institute of Technology (MIT).
He is currently a Dean’s Chair Associate Professor with the Department of Electrical and Computer Engineering and the Department of Mathematics, National University of Singapore (NUS). His research interests include information theory, machine learning, and statistical signal processing.
He was also an IEEE Information Theory Society Distinguished Lecturer in 2018/9. He is currently serving as an Associate Editor for the IEEE Transactions on Signal Processing and an Associate Editor for Machine Learning for the IEEE Transactions on Information Theory. He is a member of the IEEE Information Theory Society Board of Governors.
allie.mcfadden@cc.gatech.edu
]]>Registration is required. Register here.
Recent progress in deep generative models such as Generative Adversarial Networks (GANs) has enabled synthesizing photo-realistic images, such as faces and scenes. However, it remains much less explored on what has been learned in the deep generative representation and why diverse realistic images can be synthesized. In this talk, I will present our recent series work from GenForce (https://genforce.github.io/) on interpreting and utilizing latent space of the GANs. Identifying these semantics not only allows us to better understand the inner working of the deep generative models but also facilitates versatile image editings. I will also briefly talk about the inverse problem (how to invert a given image into the latent code) and the fairness of the generative model.
Bolei Zhou is an Assistant Professor with the Information Engineering Department at the Chinese University of Hong Kong. He earned his PhD in computer science at the Massachusetts Institute of Technology. His research is on machine perception and autonomy, with a focus on enabling interpretable human-AI interactions. He received the MIT Tech Review’s Innovators under 35 in Asia-Pacific Award, Facebook Fellowship, Microsoft Research Asia Fellowship, MIT Greater China Fellowship, and his research was featured in media outlets such as TechCrunch, Quartz, and MIT News.
Communications Officer
allie.mcfadden@cc.gatech.edu
]]>Registration is required. Register here.
Eightfold's mission statement is to find the right career for the right person. In order to successfully find the right career for any individual, it is imperative to have a deep understanding of their past and current accomplishments and use them to forecast their future capabilities and potential. The Machine Learning at Eightfold tackles this exact problem that helps to hire for potential and answer the question "What’s Next for You"
Sanjeet Hajarnis is a Principal Engineer at Eightfold AI and focuses on building scalable Machine Learning platforms at Eightfold. At Eightfold, he focuses on building large scale language models. Prior to Eightfold, Sanjeet has worked at Uber (Matching) and Facebook (News Feed). He graduated from Georgia Tech in 2012 with a specialization in Machine Learning.
]]>Communications Officer
allie.mcfadden@cc.gatech.edu
]]>
Date: April 5th
Time: 10:05 AM – 12:00 PM ET
Location: Coda C1115
Wenbo Chen
Machine Learning PhD Student
H. Milton Stewart School of Industrial and Systems Engineering
Georgia Institute of Technology
Committee
Pascal Van Hentenryck (Advisor), School of Industrial and Systems Engineering, Georgia Institute of Technology
Alan Erera, School of Industrial and Systems Engineering, Georgia Institute of Technology
Yao Xie, School of Industrial and Systems Engineering, Georgia Institute of Technology
Siva Maguluri, School of Industrial and Systems Engineering, Georgia Institute of Technology
Daniel K. Molzahn, School of Electrical and Computer Engineering, Georgia Institute of Technology
Abstract
The integration of renewable energy introduces increased uncertainties in power systems. These uncertainties bring new types of risk and motivate the Independent System Operators (ISOs) in the US to perform risk analysis in real time. However, traditional optimization-based risk assessment is not practical given the tight time budget of real-time operation as it requires systematically solving a sequence of large-scale optimization instances for thousands of load and renewable scenarios. Additionally, day-to-day operations often involve numerous instances. This cumulates a large dataset and gives the opportunity to shift the computational burden from online to offline through machine learning. These challenges and opportunities have motivated the thesis to develop optimization proxies, differentiable programs to learn the input-output mapping of underlying optimization, to enable real-time risk assessment by the principled integration of Machine Learning (ML) and optimization.
First, this thesis focuses on the practicality of developing optimization proxies for industrial-size Security-Constrained Economic Dispatch (SCED) problems, a foundational building block in US energy market clearing. Motivated by a principled analysis of the market-clearing optimization and simulation process in a realistic US energy market pipeline, the thesis proposes a novel just-in-time ML pipeline that addresses the main challenges incurred by the variability in load, renewable output, and production costs, as well as the combinatorial structure of commitment decisions. Second, the thesis presents a novel End-to-End Learning and Repair (E2ELR) architecture to unifiedly improve the feasibility and scalability. E2ELR combines deep learning with closed-form, differentiable optimization layers, thereby integrating learning and feasibility in an end-to-end fashion. The results demonstrate that the E2ELR achieves state-of-the-art performance, with optimality gaps that outperform other baselines by at least an order of magnitude. Finally, the thesis presents the first real-time risk assessment framework for large-scale power systems with high granularity e.g., at the level of generators and transmission lines.
]]>
You are cordially invited to my thesis defense on Dec 2nd.
Title: Towards Interpretable and Controllable Machine Learning Models via Logic Reasoning
Date: 12/02/2024
Time: 11:00AM EST
Location: https://gatech.zoom.us/j/94374751336
Yuan Yang
Machine Learning PhD Student
School of Computational Science and Engineering
Georgia Institute of Technology
Committee
1 Dr. Faramarz Fekri, School of Electrical and Computer Engineering, Georgia Institute of Technology (Advisor)
2 Dr. Zsolt Kira, School of Interactive Computing, Georgia Institute of Technology
3 Dr. Larry Heck, School of Electrical and Computer Engineering, Georgia Institute of Technology
4 Dr. Viveck Cadambe, School of Electrical and Computer Engineering, Georgia Institute of Technology
5 Dr. Bo Dai, School of Computational Science and Engineering, Georgia Institute of Technology
Abstract
Modern machine learning models have provided new capabilities across a spectrum of applications in vision, reasoning, and natural language processing. However, these models are criticized for being non-interpretable, data-inefficient, and vulnerable to subtle perturbations such as adversarial attacks and distribution shifts. Addressing these issues remains at the center of developing trustworthy ML systems for real-world applications.
Our research focuses on providing a principled solution to these issues through logic reasoning formalism.
Specifically, we study the fundamental technique of inductive logic programming (ILP) that learns and represents patterns in knowledge graphs as first-order logic (FOL) rules, providing an interpretable approach to various reasoning tasks on structured data:
(1) we investigate the connection between model explanation and logic formalism and propose frameworks for explaining and defending ML models via logic reasoning;
(2) we formalize logic reasoning methods as a novel data programming paradigm and propose data-efficient frameworks for model training and evaluation;
(3) to improve the expressiveness of the ILP technique, we propose to extend the model to the temporal domain and hypergraphs so that one can generalize FOL rules on complex structures
Furthermore, our research explores the integration of large language models (LLMs) with logical reasoning techniques to enhance interpretability, data efficiency, and controllability in machine learning systems. We investigate:
(1) the potential of LLMs in translating natural language to formal logical representations to solve complex reasoning problems;
(2) enhancing LLMs' reasoning capability on open-ended, ambiguous problems by incorporating formal logic reasoning, thereby improving their controllability and robustness beyond narrowly defined domains.
By combining logic reasoning with the latest advancements in LLMs, our research aims to bridge the gap between powerful ML models and the need for explainable, efficient, and reliable AI systems in real-world applications.
]]>