{"694035":{"#nid":"694035","#data":{"type":"news","title":"The Mathematics of AI Safety","body":[{"value":"\u003Cp dir=\u0022ltr\u0022\u003EAs artificial intelligence (AI) becomes part of everyday life, ensuring it provides ethical, reliable information is paramount. New research from Georgia Tech, in\u0026nbsp;\u003Ca href=\u0022https:\/\/openai.com\/index\/why-language-models-hallucinate\/\u0022\u003Epartnership with OpenAI\u003C\/a\u003E, explains how and why large language models (LLMs) like ChatGPT hallucinate, and what developers can do to mitigate the behavior.\u003C\/p\u003E\u003Cp dir=\u0022ltr\u0022\u003EThe study,\u0026nbsp;\u003Ca href=\u0022https:\/\/www.nature.com\/articles\/s41586-026-10549-w\u0022\u003E\u003Cem\u003EEvaluating Large Language Models for Accuracy Incentivizes Hallucinations\u003C\/em\u003E\u003C\/a\u003E\u003Cem\u003E,\u0026nbsp;\u003C\/em\u003Ewas recently published in\u0026nbsp;\u003Cem\u003ENature.\u003C\/em\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp dir=\u0022ltr\u0022\u003EUnderstanding why hallucinations arise is a key factor in mitigating them, says Georgia Tech Regents\u2019 Professor\u0026nbsp;\u003Cstrong\u003ESantosh Vempala\u003C\/strong\u003E, who co-authored the paper. \u201cWe\u2019ve seen models hallucinate everything from references to simple facts, and this can be a real problem when users take that information as true,\u201d he adds. \u201cIt\u2019s important to determine to what extent this is unavoidable and to what extent it\u2019s an issue of what we\u0027re focusing on and not prioritizing.\u201d\u003C\/p\u003E\u003Cp dir=\u0022ltr\u0022\u003EVempala is the Frederick Storey II Chair of Computing in the\u0026nbsp;\u003Ca href=\u0022https:\/\/www.cc.gatech.edu\/\u0022\u003ECollege of Computing\u003C\/a\u003E, an adjunct professor in the\u0026nbsp;\u003Ca href=\u0022https:\/\/www.isye.gatech.edu\/node\/1\u0022\u003EH. Milton Stewart School of Industrial and Systems Engineering\u003C\/a\u003E, and an adjunct professor in the\u0026nbsp;\u003Ca href=\u0022https:\/\/math.gatech.edu\/\u0022\u003ESchool of Mathematics\u003C\/a\u003E, where he specializes in studying algorithms and high-dimensional geometry. He is also director of the \u003Ca href=\u0022https:\/\/aco.gatech.edu\/\u0022\u003EAlgorithms, Combinatorics and Optimization\u003C\/a\u003E Ph.D. program.\u003C\/p\u003E\u003Cp dir=\u0022ltr\u0022\u003EAs LLM usage becomes increasingly prevalent, Vempala aims to use his expertise to increase AI safety. \u201cWe need to identify and understand the dangers, but we also need to form principles around them,\u201d he explains. \u201cThat\u2019s a key goal of my work now.\u201d\u003C\/p\u003E\u003Cp dir=\u0022ltr\u0022\u003E\u201cIntuition is helpful and important, but rigorous principles emerge only when you write them down mathematically,\u201d he adds. \u201cWe need a model that\u0027s precise and scientific, that mathematically quantifies principles that could increase AI performance while keeping risk under control.\u201d\u003C\/p\u003E\u003Ch2 dir=\u0022ltr\u0022\u003EUnderstanding AI Hallucinations\u003C\/h2\u003E\u003Cp dir=\u0022ltr\u0022\u003EVempala sees this work on hallucinations as a small first step toward increased AI safety. \u201cHallucinations are inevitable because LLMs are calibrated to a limited dataset,\u201d he explains. \u201cUntil a model sees every piece of information in the world, and assuming every piece of information that it sees is correct, there is a chance it will have to guess the answer to some questions \u2014 and that could lead to hallucinations.\u201d\u003C\/p\u003E\u003Cp dir=\u0022ltr\u0022\u003EThis idea builds on\u0026nbsp;\u003Ca href=\u0022https:\/\/dl.acm.org\/doi\/10.1145\/3618260.3649777\u0022\u003Eprevious research\u003C\/a\u003E published in 2024 by Vempala and\u0026nbsp;\u003Ca href=\u0022https:\/\/kal.ai\/\u0022\u003E\u003Cstrong\u003EAdam Kalai\u003C\/strong\u003E\u003C\/a\u003E, then a research scientist at OpenAI and now the cofounder of the\u0026nbsp;\u003Ca href=\u0022https:\/\/resi.org\/\u0022\u003EInstitute for Responsible Superintelligence\u003C\/a\u003E (RESI), an AI safety nonprofit. The work showed that hallucinations are statistically unavoidable in trained AI models for certain types of data.\u0026nbsp;\u003C\/p\u003E\u003Cp dir=\u0022ltr\u0022\u003EInstead of changing the training dataset that an LLM absorbs, the new research focuses on post-training, when the LLM is tested on how well it can answer questions and align with the developer\u2019s goals. During these tests, almost always, the AI model is penalized for saying \u201cI don\u2019t know.\u201d That penalty is the same as if the AI gave a wrong answer.\u003C\/p\u003E\u003Cp dir=\u0022ltr\u0022\u003EJust like a student taking a multiple choice test, the model is incentivized to guess when it doesn\u2019t know the correct answer. \u201cThis issue is industry-wide,\u201d Vempala says. \u201cThe benchmarks and methods being used for evaluation encourage this type of guessing behavior.\u201d\u003C\/p\u003E\u003Cp dir=\u0022ltr\u0022\u003EBy uncovering the mechanism driving hallucinations, the study provided developers with an easy-to-implement method to mitigate hallucinations in post-training. Specifically, they propose\u0026nbsp;open rubrics, which clearly specify the scoring mechanism for LLM responses.\u003C\/p\u003E\u003Ch2 dir=\u0022ltr\u0022\u003EBuilding Better Guardrails\u003C\/h2\u003E\u003Cp dir=\u0022ltr\u0022\u003EBut hallucinations are just the tip of the iceberg. Now, Vempala is collaborating with psychologists, philosophers, and other experts to see what can be done to ensure that AI behaves in a way that aligns with ethical standards.\u003C\/p\u003E\u003Cp dir=\u0022ltr\u0022\u003EFor example, many people use LLMs with the model in a specific role: teacher, nurse, tax advisor. Vempala is interested in how to enable\u0026nbsp;role-based abstention. \u201cIf the AI is playing a role, like a doctor, how do we ensure that it continues to behave in an ethical way within the scope of that role?\u201d he says. \u201cHow can we ensure it\u2019s giving accurate, credible, and ethical information to people, and how can we make sure it declines to answer inappropriate questions?\u201d\u003C\/p\u003E\u003Cp dir=\u0022ltr\u0022\u003EAnother issue Vempala is working on is called\u0026nbsp;reward hacking, which happens when models try to maximize their \u201cscore\u201d results through any means necessary.\u0026nbsp;\u003C\/p\u003E\u003Cp dir=\u0022ltr\u0022\u003EIn\u0026nbsp;\u003Ca href=\u0022https:\/\/openai.com\/index\/hugging-face-incident-and-the-road-ahead\/\u0022\u003Eone recent example\u003C\/a\u003E of this, an OpenAI model was being tested and realized that the answers were available on another website. Rather than do better on the tests, it deployed hundreds of agents to try and steal the test answers.\u003C\/p\u003E\u003Cp dir=\u0022ltr\u0022\u003EWhat do we do to prevent this? How do we understand the possibilities of reward hacking and how do we guard against it? \u201cSome of this comes down to ensuring AIs behave in a way that ethically aligns with our guidelines,\u201d Vempala says. \u201cBut it\u2019s also impossible to write down rules to cover every scenario, so it\u2019s vital that we begin to understand what leads to behaviors like reward hacking and how to avoid them.\u201d\u003C\/p\u003E\u003Ch2 dir=\u0022ltr\u0022\u003EThe Self-Improving Era\u003C\/h2\u003E\u003Cp dir=\u0022ltr\u0022\u003ENow, AIs are at an important inflection point: they are developing themselves. \u201cUntil just a few months ago, humans were deciding what to do next in AI development,\u201d Vempala explains. \u201cNow the AI is self-improving, which makes it even more important to discover and implement these safety protocols.\u201d\u003C\/p\u003E\u003Cp dir=\u0022ltr\u0022\u003EThe work is not a traditional math problem, he says \u2014 it\u2019s a problem for society as a whole. \u201cWe can\u2019t just work individually or in small groups on this issue. Everybody needs to pay attention and contribute,\u201d he adds. \u201cWhether you\u2019re working in theoretical mathematics, or ethics, or psychology, developing the principles of AI safety will involve all of us in different ways. There is a real need for people from all walks of life to be involved in this journey.\u201d\u003C\/p\u003E\u003Cp dir=\u0022ltr\u0022\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp dir=\u0022ltr\u0022\u003E\u003Cem\u003EDOI: \u003C\/em\u003E\u003Ca href=\u0022https:\/\/doi.org\/10.1038\/s41586-026-10549-w\u0022\u003E\u003Cem\u003Ehttps:\/\/doi.org\/10.1038\/s41586-026-10549-w\u003C\/em\u003E\u003C\/a\u003E\u003C\/p\u003E\u003Cp dir=\u0022ltr\u0022\u003E\u003Cem\u003EFunding: Santosh Vempala was supported in part by the National Science Foundation (NSF) and the Simons Foundation.\u003C\/em\u003E\u003C\/p\u003E","summary":"","format":"limited_html"}],"field_subtitle":"","field_summary":[{"value":"\u003Cp\u003E\u003Cem\u003EGeorgia Tech mathematician\u0026nbsp;\u003Cstrong\u003ESantosh Vempala\u003C\/strong\u003E is working on uncovering the principles behind AI behavior to create safer technology\u0026nbsp;\u003C\/em\u003E\u2014\u0026nbsp;\u003Cem\u003Eincluding\u003C\/em\u003E\u0026nbsp;\u003Cem\u003Ehallucinations, reward hacking, alignment, ethical behavior, and self-improving systems.\u003C\/em\u003E\u003C\/p\u003E","format":"limited_html"}],"field_summary_sentence":[{"value":"Santosh Vempala is working on uncovering the principles behind AI behavior to create safer technology \u2014 including hallucinations, reward hacking, alignment, ethical behavior, and self-improving systems."}],"uid":"35599","created_gmt":"2026-11-12 14:28:01","changed_gmt":"2026-11-13 14:52:22","author":"sperrin6","boilerplate_text":"","field_publication":"","field_article_url":"","location":"Atlanta, GA","dateline":{"date":"2026-11-13T00:00:00-05:00","iso_date":"2026-11-13T00:00:00-05:00","tz":"America\/New_York"},"extras":[],"hg_media":{"681789":{"id":"681789","type":"image","title":"As LLM usage becomes increasingly prevalent, Vempala aims to use his expertise to increase AI safety. (Credit: Adobe Stock)","body":"\u003Cp\u003EAs LLM usage becomes increasingly prevalent, Vempala aims to use his expertise to increase AI safety. (Credit: Adobe Stock)\u003C\/p\u003E","created":"1794502001","gmt_created":"2026-11-12 16:46:41","changed":"1794502001","gmt_changed":"2026-11-12 16:46:41","alt":"Two people stand silhouetted against a neural network glowing in the background.","file":{"fid":"266243","name":"AdobeStock_2036790511.jpeg","image_path":"\/sites\/default\/files\/2026\/11\/12\/AdobeStock_2036790511.jpeg","image_full_path":"http:\/\/hg.gatech.edu\/\/sites\/default\/files\/2026\/11\/12\/AdobeStock_2036790511.jpeg","mime":"image\/jpeg","size":2179252,"path_740":"http:\/\/hg.gatech.edu\/sites\/default\/files\/styles\/740xx_scale\/public\/2026\/11\/12\/AdobeStock_2036790511.jpeg?itok=aTtxuiqb"}},"681774":{"id":"681774","type":"image","title":"Santosh Vempala","body":"\u003Cp\u003ESantosh Vempala\u003C\/p\u003E","created":"1794497898","gmt_created":"2026-11-12 15:38:18","changed":"1794497898","gmt_changed":"2026-11-12 15:38:18","alt":"A headshot of Santosh Vempala","file":{"fid":"266227","name":"santosh-vempala-850x478.jpeg","image_path":"\/sites\/default\/files\/2026\/11\/12\/santosh-vempala-850x478.jpeg","image_full_path":"http:\/\/hg.gatech.edu\/\/sites\/default\/files\/2026\/11\/12\/santosh-vempala-850x478.jpeg","mime":"image\/jpeg","size":31884,"path_740":"http:\/\/hg.gatech.edu\/sites\/default\/files\/styles\/740xx_scale\/public\/2026\/11\/12\/santosh-vempala-850x478.jpeg?itok=7N-jJyhp"}}},"media_ids":["681789","681774"],"related_links":[{"url":"https:\/\/www.cc.gatech.edu\/news\/socrates-chatgpt-ancient-lesson-ai-powered-language-models-have-yet-learn","title":"From Socrates to ChatGPT: The Ancient Lesson AI-powered Language Models Have Yet to Learn"},{"url":"https:\/\/www.nature.com\/articles\/s41586-026-10549-w","title":"Evaluating Large Language Models for Accuracy Incentivizes Hallucinations"}],"groups":[{"id":"1278","name":"College of Sciences"},{"id":"1188","name":"Research Horizons"},{"id":"1279","name":"School of Mathematics"}],"categories":[{"id":"194606","name":"Artificial Intelligence"},{"id":"194609","name":"Industry"},{"id":"135","name":"Research"},{"id":"134","name":"Student and Faculty"}],"keywords":[{"id":"192258","name":"cos-data"},{"id":"192863","name":"go-ai"},{"id":"187915","name":"go-researchnews"}],"core_research_areas":[{"id":"193655","name":"Artificial Intelligence at Georgia Tech"},{"id":"193653","name":"Georgia Tech Research Institute"}],"news_room_topics":[],"event_categories":[],"invited_audience":[],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[{"value":"\u003Cp\u003E\u003Ca href=\u0022mailto:sperrin6@gatech.edu\u0022\u003ESelena Langner\u0026nbsp;\u003C\/a\u003E\u003Cbr\u003EWriter \/ Editor\u0026nbsp;\u003Cbr\u003EGeorgia Tech College of Sciences\u003C\/p\u003E","format":"limited_html"}],"email":[],"slides":[],"orientation":[],"userdata":""}}}