Featuring Scott Yih, Facebook AI Research (FAIR)

Abstract: While large-scale language models work incredibly well, it is expensive to train them, difficult to explain their predictions, and nearly impossible to keep them current over time. It is unclear when we can trust their predictions, and none of the current large language models can answer questions about current topics, such as COVID-19, since the corpora used for their training were created several years ago. To develop the next generation of general-purpose language models with smaller, simpler, and much more efficient models, we believe information retrieval is a key component. When interacting with each other and with the world, humans tap into many different forms of knowledge, including world knowledge (e.g., commonsense, updated world facts, trending news) and user knowledge (e.g., conversational memory, social interactions, additional context such as location, etc.). To incorporate this capability in AI applications, information retrieval provides models access to (potentially large) collections of documents that can contain such knowledge. Specifically, we envision that the complete system consists of a small, core model that can easily access additional, task-related knowledge via retrieval, and perform comparably to the largest language models available today. 

In this talk, I will first give a research overview of retrieval-augmented language models. Then, I will share some of our recent work, including a general framework that improves any language models by adding a retrieval component, as well as how we apply instruction tuning to both the language model and retrieval system to further increase the gain. Finally, I'll conclude the talk by discussing some of the lessons we learned and the problems we plan to address in the near future. 

Bio: Scott Wen-tau Yih is a Research Scientist at FAIR, Meta. His research interests include natural language processing, machine learning and information retrieval. Before joining Meta, Yih was a Principal Research Scientist at the Allen Institute for Artificial Intelligence (AI2), working on scientific question answering. Prior to that, Yih had spent 12 years at Microsoft Research, working on a variety of projects including email spam filtering, keyword extraction and search & ad relevance. His recent work focuses on continuous representations and neural models for question answering and retrieval; some of his well-known work includes WikiQA, RAG and DPR. Yih received the best paper award from CoNLL’11, an outstanding paper award from ACL’15 and has served as program co-chairs (CEAS’09, CoNLL’14, EMNLP’21) and senior area chairs for NLP (ACL, NNACL, EMNLP, EACL) and ML (ICLR, NeurIPS) conferences.