{"682320":{"#nid":"682320","#data":{"type":"news","title":" Forensics Tool \u2018Reanimates\u2019 the \u2018Brains\u2019 of AIs That Fail in Order to Understand What Went Wrong","body":[{"value":"\u003Cdiv class=\u0022theconversation-article-body\u0022\u003E\u003Cp\u003EFrom drones delivering medical supplies to digital assistants performing everyday tasks, AI-powered systems are becoming increasingly embedded in everyday life. The creators of these innovations promise transformative benefits. For some people, mainstream applications such as ChatGPT and Claude can seem like magic. But these systems are not magical, nor are they foolproof \u2013 they can and do regularly fail to work as intended.\u003C\/p\u003E\u003Cp\u003EAI systems can malfunction due to technical design flaws or biased training data. They can also suffer from vulnerabilities in their code, which can be exploited by malicious hackers. Isolating the cause of an AI failure is imperative for fixing the system.\u003C\/p\u003E\u003Cp\u003EBut AI systems are typically opaque, even to their creators. The challenge is how to investigate AI systems after they fail or fall victim to attack. There are techniques for inspecting AI systems, but they require access to the AI system\u2019s internal data. This access is not guaranteed, especially to forensic investigators called in to determine the cause of a proprietary AI system failure, making investigation impossible.\u003C\/p\u003E\u003Cp\u003EWe are \u003Ca href=\u0022https:\/\/scholar.google.com\/citations?user=zzJmhKIAAAAJ\u0026amp;hl=enough\u0022\u003Ecomputer scientists\u003C\/a\u003E \u003Ca href=\u0022https:\/\/scholar.google.com\/citations?hl=en\u0026amp;user=1GsJvtwAAAAJ\u0026amp;view_op=list_works\u0026amp;sortby=pubdate\u0022\u003Ewho study\u003C\/a\u003E digital forensics. Our team at the Georgia Institute of Technology has built a system, \u003Ca href=\u0022https:\/\/www.usenix.org\/conference\/usenixsecurity24\/presentation\/oygenblik\u0022\u003EAI Psychiatry\u003C\/a\u003E, or AIP, that can recreate the scenario in which an AI failed in order to determine what went wrong. The system addresses the challenges of AI forensics by recovering and \u201creanimating\u201d a suspect AI model so it can be systematically tested.\u003C\/p\u003E\u003Ch2\u003EUncertainty of AI\u003C\/h2\u003E\u003Cp\u003EImagine a self-driving car veers off the road for no easily discernible reason and then crashes. Logs and sensor data might suggest that a faulty camera caused the AI to misinterpret a road sign as a command to swerve. After a mission-critical failure such as an \u003Ca href=\u0022https:\/\/www.theguardian.com\/technology\/2024\/apr\/26\/tesla-autopilot-fatal-crash\u0022\u003Eautonomous vehicle crash\u003C\/a\u003E, investigators need to determine exactly what caused the error.\u003C\/p\u003E\u003Cp\u003EWas the crash triggered by a malicious attack on the AI? In this hypothetical case, the camera\u2019s faultiness could be the result of a security vulnerability or bug in its software that was exploited by a hacker. If investigators find such a vulnerability, they have to determine whether that caused the crash. But making that determination is no small feat.\u003C\/p\u003E\u003Cp\u003EAlthough there are forensic methods for recovering some evidence from failures of drones, autonomous vehicles and other so-called cyber-physical systems, none can capture the clues required to fully investigate the AI in that system. Advanced AIs can even \u003Ca href=\u0022https:\/\/doi.org\/10.48550\/arXiv.1802.02871\u0022\u003Eupdate their decision-making\u003C\/a\u003E \u2013 and consequently the clues \u2013 continuously, making it impossible to investigate the most up-to-date models with existing methods.\u003C\/p\u003E\u003Cfigure\u003E\u003Cp\u003E\u003Ciframe width=\u0022440\u0022 height=\u0022260\u0022 src=\u0022https:\/\/www.youtube.com\/embed\/PcfXjfyPDgE?wmode=transparent\u0026amp;start=0\u0022 frameborder=\u00220\u0022 allowfullscreen=\u0022\u0022\u003E\u003C\/iframe\u003E\u003C\/p\u003E\u003Cfigcaption\u003E\u003Cspan class=\u0022caption\u0022\u003EResearchers are working on making AI systems more transparent, but unless and until those efforts transform the field, there will be a need for forensics tools to at least understand AI failures.\u003C\/span\u003E\u003C\/figcaption\u003E\u003C\/figure\u003E\u003Ch2\u003EPathology for AI\u003C\/h2\u003E\u003Cp\u003EAI Psychiatry applies a series of forensic algorithms to isolate the data behind the AI system\u2019s decision-making. These pieces are then reassembled into a functional model that performs identically to the original model. Investigators can \u201creanimate\u201d the AI in a controlled environment and test it with malicious inputs to see whether it exhibits harmful or hidden behaviors.\u003C\/p\u003E\u003Cp\u003EAI Psychiatry takes in as input \u003Ca href=\u0022https:\/\/www.techtarget.com\/whatis\/definition\/memory-dump\u0022\u003Ea memory image\u003C\/a\u003E, a snapshot of the bits and bytes loaded when the AI was operational. The memory image at the time of the crash in the autonomous vehicle scenario holds crucial clues about the internal state and decision-making processes of the AI controlling the vehicle. With AI Psychiatry, investigators can now lift the exact AI model from memory, dissect its bits and bytes, and load the model into a secure environment for testing.\u003C\/p\u003E\u003Cp\u003EOur team tested AI Psychiatry on 30 AI models, 24 of which were intentionally \u201c\u003Ca href=\u0022https:\/\/csrc.nist.gov\/glossary\/term\/backdoor\u0022\u003Ebackdoored\u003C\/a\u003E\u201d to produce incorrect outcomes under specific triggers. The system was successfully able to recover, rehost and test every model, including models commonly used in real-world scenarios such as street sign recognition in autonomous vehicles.\u003C\/p\u003E\u003Cp\u003EThus far, our tests suggest that AI Psychiatry can effectively solve the digital mystery behind a failure such as an autonomous car crash that previously would have left more questions than answers. And if it does not find a vulnerability in the car\u2019s AI system, AI Psychiatry allows investigators to rule out the AI and look for other causes such as a faulty camera.\u003C\/p\u003E\u003Ch2\u003ENot Just for Autonomous Vehicles\u003C\/h2\u003E\u003Cp\u003EAI Psychiatry\u2019s main algorithm is generic: It focuses on the universal components that all AI models must have to make decisions. This makes our approach readily extendable to any AI models that use popular AI development frameworks. Anyone working to investigate a possible AI failure can use our system to assess a model without prior knowledge of its exact architecture.\u003C\/p\u003E\u003Cp\u003EWhether the AI is a bot that makes product recommendations or a system that guides autonomous drone fleets, AI Psychiatry can recover and rehost the AI for analysis. AI Psychiatry is \u003Ca href=\u0022https:\/\/github.com\/CyFI-Lab-Public\/AiP\u0022\u003Eentirely open source\u003C\/a\u003E for any investigator to use.\u003C\/p\u003E\u003Cp\u003EAI Psychiatry can also serve as a valuable tool for conducting audits on AI systems before problems arise. With government agencies from law enforcement to child protective services integrating AI systems into their workflows, AI audits are becoming an increasingly common oversight requirement at the state level. With a tool like AI Psychiatry in hand, auditors can apply a consistent forensic methodology across diverse AI platforms and deployments.\u003C\/p\u003E\u003Cp\u003EIn the long run, this will pay meaningful dividends both for the creators of AI systems and everyone affected by the tasks they perform.\u003C!-- Below is The Conversation\u0027s page counter tag. Please DO NOT REMOVE. --\u003E\u003Cimg style=\u0022border-color:!important;border-style:none;box-shadow:none !important;margin:0 !important;max-height:1px !important;max-width:1px !important;min-height:1px !important;min-width:1px !important;opacity:0 !important;outline:none !important;padding:0 !important;\u0022 src=\u0022https:\/\/counter.theconversation.com\/content\/247769\/count.gif?distributor=republish-lightbox-basic\u0022 alt=\u0022The Conversation\u0022 width=\u00221\u0022 height=\u00221\u0022 referrerpolicy=\u0022no-referrer-when-downgrade\u0022\u003E\u003C!-- End of code. If you don\u0027t see any code above, please get new code from the Advanced tab after you click the republish button. The page counter does not collect any personal data. More info: https:\/\/theconversation.com\/republishing-guidelines --\u003E\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cem\u003EThis article is republished from \u003C\/em\u003E\u003Ca href=\u0022https:\/\/theconversation.com\u0022\u003E\u003Cem\u003EThe Conversation\u003C\/em\u003E\u003C\/a\u003E\u003Cem\u003E under a Creative Commons license. Read the \u003C\/em\u003E\u003Ca href=\u0022https:\/\/theconversation.com\/forensics-tool-reanimates-the-brains-of-ais-that-fail-in-order-to-understand-what-went-wrong-247769\u0022\u003E\u003Cem\u003Eoriginal article\u003C\/em\u003E\u003C\/a\u003E\u003Cem\u003E.\u003C\/em\u003E\u003C\/p\u003E\u003C\/div\u003E","summary":"","format":"full_html"}],"field_subtitle":"","field_summary":[{"value":"\u003Cp\u003EAI-powered systems are not magical, nor are they foolproof \u2013 they can and do regularly fail to work as intended.\u003C\/p\u003E","format":"limited_html"}],"field_summary_sentence":[{"value":"AI-powered systems are not magical, nor are they foolproof \u2013 they can and do regularly fail to work as intended."}],"uid":"27469","created_gmt":"2025-04-30 18:06:13","changed_gmt":"2026-03-19 13:16:46","author":"Kristen Bailey","boilerplate_text":"","field_publication":"","field_article_url":"","location":"Atlanta, GA","dateline":{"date":"2025-04-30T00:00:00-04:00","iso_date":"2025-04-30T00:00:00-04:00","tz":"America\/New_York"},"extras":[],"hg_media":{"677057":{"id":"677057","type":"image","title":"Tesla crashes are only the most glaring of AI failures. South Jordan Police Department via APPEAR","body":"\u003Cp\u003ETesla crashes are only the most glaring of AI failures. \u003Ca href=\u0022https:\/\/newsroom.ap.org\/detail\/TeslaCrashUtah\/e4e84ea27288453ba6950d92d412b2d7\/photo\u0022\u003ESouth Jordan Police Department via APPEAR\u003C\/a\u003E\u003C\/p\u003E","created":"1746814313","gmt_created":"2025-05-09 18:11:53","changed":"1746814313","gmt_changed":"2025-05-09 18:11:53","alt":"Tesla crashes are only the most glaring of AI failures. South Jordan Police Department via APPEAR","file":{"fid":"260919","name":"file-20250429-56-y0chq7.jpg","image_path":"\/sites\/default\/files\/2025\/05\/09\/file-20250429-56-y0chq7.jpg","image_full_path":"http:\/\/hg.gatech.edu\/\/sites\/default\/files\/2025\/05\/09\/file-20250429-56-y0chq7.jpg","mime":"image\/jpeg","size":824137,"path_740":"http:\/\/hg.gatech.edu\/sites\/default\/files\/styles\/740xx_scale\/public\/2025\/05\/09\/file-20250429-56-y0chq7.jpg?itok=zaQLBcSx"}}},"media_ids":["677057"],"related_links":[{"url":"https:\/\/theconversation.com\/forensics-tool-reanimates-the-brains-of-ais-that-fail-in-order-to-understand-what-went-wrong-247769","title":"Read This Article on The Conversation"}],"groups":[{"id":"658168","name":"Experts"},{"id":"1214","name":"News Room"},{"id":"1188","name":"Research Horizons"},{"id":"1255","name":"School of Electrical and Computer Engineering"}],"categories":[],"keywords":[{"id":"187915","name":"go-researchnews"}],"core_research_areas":[],"news_room_topics":[{"id":"71881","name":"Science and Technology"}],"event_categories":[],"invited_audience":[],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[{"value":"\u003Ch5\u003EAuthor:\u003C\/h5\u003E\u003Cp\u003E\u003Ca href=\u0022https:\/\/theconversation.com\/profiles\/david-oygenblik-2299577\u0022\u003EDavid Oygenblik\u003C\/a\u003E, Ph.D. Student in Electrical and Computer Engineering, Georgia Institute of Technology\u003C\/p\u003E\u003Cp\u003E\u003Ca href=\u0022https:\/\/theconversation.com\/profiles\/brendan-saltaformaggio-2299579\u0022\u003EBrendan Saltaformaggio\u003C\/a\u003E, Associate Professor of Cybersecurity and Privacy, and Electrical and Computer Engineering, Georgia Institute of Technology\u003C\/p\u003E\u003Ch5\u003EMedia Contact:\u003C\/h5\u003E\u003Cp\u003EShelley Wunder-Smith\u003Cbr\u003E\u003Ca href=\u0022mailto:shelley.wunder-smith@research.gatech.edu\u0022\u003Eshelley.wunder-smith@research.gatech.edu\u003C\/a\u003E\u003C\/p\u003E","format":"limited_html"}],"email":[],"slides":[],"orientation":[],"userdata":""}}}