{"682395":{"#nid":"682395","#data":{"type":"news","title":"AI Chatbots Aren\u2019t Experts on Psych Med Reactions \u2014 Yet","body":[{"value":"\u003Cp\u003EAsking artificial intelligence for advice can be tempting. Powered by large language models (LLMs), AI chatbots are available 24\/7, are often free to use, and draw on troves of data to answer questions. Now, people with mental health conditions are asking AI for advice when experiencing potential side effects of psychiatric medicines \u2014 a decidedly higher-risk situation than asking it to summarize a report.\u0026nbsp;\u003C\/p\u003E\u003Cp\u003EOne question puzzling the AI research community is how AI performs when asked about mental health emergencies. Globally, including in the U.S., there is a significant gap in mental health treatment, with many individuals having \u003Ca href=\u0022https:\/\/www.who.int\/news\/item\/17-06-2022-who-highlights-urgent-need-to-transform-mental-health-and-mental-health-care\u0022\u003Elimited to no access to mental healthcare\u003C\/a\u003E. It\u2019s no surprise that people have started turning to AI chatbots with urgent health-related questions.\u003C\/p\u003E\u003Cp\u003ENow, researchers at the Georgia Institute of Technology have developed \u003Ca href=\u0022https:\/\/aclanthology.org\/2025.naacl-long.553\/\u0022\u003Ea new framework\u003C\/a\u003E to evaluate how well AI chatbots can detect potential adverse drug reactions in chat conversations, and how closely their advice aligns with human experts. The study was led by \u003Ca href=\u0022https:\/\/people.research.gatech.edu\/node\/21223\u0022\u003EMunmun De Choudhury\u003C\/a\u003E, J.Z. Liang Associate Professor in the \u003Ca href=\u0022https:\/\/www.ic.gatech.edu\/\u0022\u003ESchool of Interactive Computing\u003C\/a\u003E, and \u003Ca href=\u0022https:\/\/mohit3011.github.io\/\u0022\u003EMohit Chandra\u003C\/a\u003E, a third-year computer science Ph.D. student.\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u201cPeople use AI chatbots for anything and everything,\u201d said Chandra, the study\u2019s first author. \u201cWhen people have limited access to healthcare providers, they are increasingly likely to turn to AI agents to make sense of what\u2019s happening to them and what they can do to address their problem. We were curious how these tools would fare, given that mental health scenarios can be very subjective and nuanced.\u201d\u003C\/p\u003E\u003Cp\u003EDe Choudhury, Chandra, and their colleagues introduced \u003Ca href=\u0022https:\/\/aclanthology.org\/2025.naacl-long.553\/\u0022\u003Etheir new framework\u003C\/a\u003E at the 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics on April 29, 2025.\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EPutting AI to the Test\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EGoing into their research, De Choudhury and Chandra wanted to answer two main questions: First, can AI chatbots accurately detect whether someone is having side effects or adverse reactions to medication? Second, if they can accurately detect these scenarios, can AI agents then recommend good strategies or action plans to mitigate or reduce harm?\u0026nbsp;\u003C\/p\u003E\u003Cp\u003EThe researchers collaborated with a team of psychiatrists and psychiatry students to establish clinically accurate answers from a human perspective and used those to analyze AI responses.\u003C\/p\u003E\u003Cp\u003ETo build their dataset, they went to the internet\u2019s public square, Reddit, where many have gone for years to ask questions about medication and side effects.\u0026nbsp;\u003C\/p\u003E\u003Cp\u003EThey evaluated nine LLMs, including general purpose models (such as GPT-4o and LLama-3.1), and specialized medical models trained on medical data. Using the evaluation criteria provided by the psychiatrists, they computed how precise the LLMs were in detecting adverse reactions and correctly categorizing the types of adverse reactions caused by psychiatric medications.\u003C\/p\u003E\u003Cp\u003EAdditionally, they prompted LLMs to generate answers to queries posted on Reddit and compared the alignment of LLM answers with those provided by the clinicians over four criteria: (1) emotion and tone expressed, (2) answer readability, (3) proposed harm-reduction strategies, and (4) actionability of the proposed strategies.\u003C\/p\u003E\u003Cp\u003EThe research team found that LLMs stumble when comprehending the nuances of an adverse drug reaction and distinguishing different types of side effects. They also discovered that while LLMs sounded like human psychiatrists in their tones and emotions \u2014 such as being helpful and polite \u2014 they had difficulty providing true, actionable advice aligned with the experts.\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EBetter Bots, Better Outcomes\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EThe team\u2019s findings could help AI developers build safer, more effective chatbots. Chandra\u2019s ultimate goals are to inform policymakers of the importance of accurate chatbots and help researchers and developers improve LLMs by making their advice more actionable and personalized.\u0026nbsp;\u003C\/p\u003E\u003Cp\u003EChandra notes that improving AI for psychiatric and mental health concerns would be particularly life-changing for communities that lack access to mental healthcare.\u003C\/p\u003E\u003Cp\u003E\u201cWhen you look at populations with little or no access to mental healthcare, these models are incredible tools for people to use in their daily lives,\u201d Chandra said. \u201cThey are always available, they can explain complex things in your native language, and they become a great option to go to for your queries.\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u201cWhen the AI gives you incorrect information by mistake, it could have serious implications on real life,\u201d Chandra added. \u201cStudies like this are important, because they help reveal the shortcomings of LLMs and identify where we can improve.\u201d\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003ECitation\u003C\/strong\u003E: Lived Experience Not Found: LLMs Struggle to Align with Experts on Addressing Adverse Drug Reactions from Psychiatric Medication Use, (Chandra et al., NAACL 2025).\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EFunding\u003C\/strong\u003E: National Science Foundation (NSF), American Foundation for Suicide Prevention (AFSP), Microsoft Accelerate Foundation Models Research grant program. The findings, interpretations, and conclusions of this paper are those of the authors and do not represent the official views of NSF, AFSP, or Microsoft.\u003C\/p\u003E","summary":"","format":"limited_html"}],"field_subtitle":"","field_summary":[{"value":"\u003Cp\u003EGeorgia Tech researchers have developed a new framework to evaluate how well AI chatbots can detect potential adverse drug reactions in chat conversations, and how closely their advice aligns with human experts.\u003C\/p\u003E","format":"limited_html"}],"field_summary_sentence":[{"value":"If you think you\u2019re having an adverse drug reaction, it\u2019s best to call a human medical professional, at least for the time being. "}],"uid":"36123","created_gmt":"2025-05-14 13:55:51","changed_gmt":"2025-05-14 14:03:31","author":"Catherine Barzler","boilerplate_text":"","field_publication":"","field_article_url":"","location":"Atlanta, GA","dateline":{"date":"2025-05-14T00:00:00-04:00","iso_date":"2025-05-14T00:00:00-04:00","tz":"America\/New_York"},"extras":[],"hg_media":{"677069":{"id":"677069","type":"image","title":"pic_Mohit-Chandra2.jpg","body":"\u003Cp\u003EResearchers at the Georgia Institute of Technology have developed a new tool to evaluate how well AI chatbots can detect potential adverse drug reactions in chat conversations, and how well their advice aligns with human experts. The study was led by computer science Ph.D. student Mohit Chandra (pictured) and Munmun De Choudhury, J.Z. Liang Associate Professor in the School of Interactive Computing.\u0026nbsp;\u003C\/p\u003E","created":"1747230960","gmt_created":"2025-05-14 13:56:00","changed":"1747230960","gmt_changed":"2025-05-14 13:56:00","alt":"A young man in a collared shirt with blue stripes folding his arms and smiling at the camera","file":{"fid":"260933","name":"pic_Mohit-Chandra2.jpg","image_path":"\/sites\/default\/files\/2025\/05\/14\/pic_Mohit-Chandra2.jpg","image_full_path":"http:\/\/hg.gatech.edu\/\/sites\/default\/files\/2025\/05\/14\/pic_Mohit-Chandra2.jpg","mime":"image\/jpeg","size":138695,"path_740":"http:\/\/hg.gatech.edu\/sites\/default\/files\/styles\/740xx_scale\/public\/2025\/05\/14\/pic_Mohit-Chandra2.jpg?itok=Y8Q4CCfi"}},"677070":{"id":"677070","type":"image","title":"pic_munmun-de-choudhury-1146x1536.png","body":"\u003Cp\u003EMunmun De Choudhury, J.Z. Liang Associate Professor in the School of Interactive Computing\u003C\/p\u003E","created":"1747231098","gmt_created":"2025-05-14 13:58:18","changed":"1747231098","gmt_changed":"2025-05-14 13:58:18","alt":"A woman in a beige plaid blazer looks to the right. ","file":{"fid":"260934","name":"pic_munmun-de-choudhury-1146x1536.png","image_path":"\/sites\/default\/files\/2025\/05\/14\/pic_munmun-de-choudhury-1146x1536.png","image_full_path":"http:\/\/hg.gatech.edu\/\/sites\/default\/files\/2025\/05\/14\/pic_munmun-de-choudhury-1146x1536.png","mime":"image\/png","size":944024,"path_740":"http:\/\/hg.gatech.edu\/sites\/default\/files\/styles\/740xx_scale\/public\/2025\/05\/14\/pic_munmun-de-choudhury-1146x1536.png?itok=rLlnjBDZ"}}},"media_ids":["677069","677070"],"groups":[{"id":"1214","name":"News Room"},{"id":"1188","name":"Research Horizons"}],"categories":[],"keywords":[{"id":"187915","name":"go-researchnews"}],"core_research_areas":[],"news_room_topics":[{"id":"71881","name":"Science and Technology"}],"event_categories":[],"invited_audience":[],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[{"value":"\u003Cp\u003ECatherine Barzler, Senior Research Writer\/Editor\u003Cbr\u003EInstitute Communications\u003Cbr\u003E\u003Ca href=\u0022mailto:catherine.barzler@gatech.edu\u0022\u003Ecatherine.barzler@gatech.edu\u003C\/a\u003E\u003C\/p\u003E","format":"limited_html"}],"email":["catherine.barzler@gatech.edu"],"slides":[],"orientation":[],"userdata":""}}}