{"689478":{"#nid":"689478","#data":{"type":"event","title":"PhD Defense by Adam Coscia","body":[{"value":"\u003Cp\u003E\u003Cstrong\u003ETitle:\u003C\/strong\u003E\u0026nbsp;Detecting and Mitigating Pedagogical Risks in Large Language Models With Visual Analytics\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EAdam Coscia\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EPh.D. Candidate in Human-Centered Computing\u0026nbsp;\u003C\/p\u003E\u003Cp\u003ESchool of Interactive Computing, College of Computing\u003C\/p\u003E\u003Cp\u003EGeorgia Institute of Technology\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Ca href=\u0022https:\/\/adamcoscia.com\/\u0022\u003Ehttps:\/\/adamcoscia.com\/\u003C\/a\u003E\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EDate:\u003C\/strong\u003E\u0026nbsp;Monday, April 27, 2026\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003ETime:\u003C\/strong\u003E\u0026nbsp;10AM - 12PM Eastern time (U.S.)\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003ELocation:\u0026nbsp;\u003C\/strong\u003E\u003Ca href=\u0022https:\/\/maps.app.goo.gl\/i8yusVJT3cryf5yC7\u0022 title=\u0022https:\/\/maps.app.goo.gl\/i8yusVJT3cryf5yC7\u0022\u003ETSRB\u003C\/a\u003E\u0026nbsp;room 334 (VIS Lab) \u2013 just walk in, show your BuzzCard to the concierge if asked\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EVirtual Meeting (hybrid):\u003C\/strong\u003E\u0026nbsp;\u003Ca href=\u0022https:\/\/gatech.zoom.us\/j\/3100254613?pwd=QWlKajNkOWlPbWkxR3N5MkZsTE9FZz09\u0022\u003Ehttps:\/\/gatech.zoom.us\/j\/3100254613?pwd=QWlKajNkOWlPbWkxR3N5MkZsTE9FZz09\u003C\/a\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003ECommittee\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EDr. Alex Endert - School of Interactive Computing, Georgia Institute of Technology\u003C\/p\u003E\u003Cp\u003EDr. Duen Horng (Polo) Chau\u0026nbsp;- School of Computational Science \u0026amp; Engineering, Georgia Institute of Technology\u003C\/p\u003E\u003Cp\u003EDr. Cindy Xiong Bearfield - School of Interactive Computing, Georgia Institute of Technology\u003C\/p\u003E\u003Cp\u003EDr. Yalong Yang\u0026nbsp;- School of Interactive Computing, Georgia Institute of Technology\u003C\/p\u003E\u003Cp\u003EDr. Scott Crossley - Department of Special Education, Vanderbilt University\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EAbstract\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EThe advent of powerful new large language models (LLMs) has catalyzed a surge in LLM-powered educational technologies, enabling transformational advances that can empower learner agency, deliver personalized study materials, and promote active learning. Yet persistent pedagogical risks, from bias and hallucinations to unfair grading and misalignment with instructional goals, highlight a critical technology gap. Existing tools for selecting, fine-tuning, and evaluating LLMs are not designed to address the unique challenges of educational contexts, making it difficult for data scientists to detect and mitigate the potential pedagogical risks of prematurely deploying LLM-powered educational technology.\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003EThis thesis addresses this gap by introducing \u003Cstrong\u003Ehuman-in-the-loop visual analytics approaches\u003C\/strong\u003E\u0026nbsp;that integrate automated analysis with interactive visualizations, enabling data scientists to more effectively discover, understand, and address pedagogical risks throughout the LLM development lifecycle. We organize these contributions under four main thrusts:\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E(1) \u003Cstrong\u003EUncovering Harmful Biases and Stereotypes in LLM Selection\u003C\/strong\u003E: We introduce KnowledgeVIS, a visual analytics system that enables interactive exploration of fill-in-the-blank prompts to surface latent biases, stereotypes, and learned associations in foundation LLMs. By supporting comparative analysis across models, KnowledgeVIS empowers data scientists to make more informed model selection decisions prior to deployment, revealing risks that are often obscured by traditional benchmark-driven evaluation.\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E(2) \u003Cstrong\u003EDiagnosing LLM Decision-Making During Fine-Tuning\u003C\/strong\u003E: We present iScore, a human-in-the-loop visual analytics system for interpreting how LLMs make scoring decisions in educational tasks such as automatic writing assessment. By linking internal model representations with input perturbations and output variations, iScore enables data scientists to diagnose unintended decision-making criteria, uncover model sensitivities, and iteratively refine fine-tuning strategies to better align with pedagogical objectives.\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E(3) \u003Cstrong\u003EMeasuring and Visualizing LLM Trustworthiness in Evaluation\u003C\/strong\u003E: We propose a novel framework for operationalizing LLM trustworthiness as a set of interpretable, pedagogically grounded metrics, coupled with visualizations that make these risks traceable within model outputs. Through a co-designed evaluation workflow, we demonstrate how these metrics improve the consistency, transparency, and defensibility of expert decision-making, while surfacing complex trade-offs that cannot be captured by traditional performance measures alone.\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E(4) \u003Cstrong\u003EBroadening Access to Visual Analytics in Education\u003C\/strong\u003E: We contribute TrustyVis, an open-source Python library that encapsulates the visual analytics techniques developed in this thesis into modular, reusable components. By lowering the barrier to integrating interactive visualizations into existing machine learning workflows, TrustyVis enables scalable and practical adoption of human-in-the-loop approaches for evaluating and improving LLM-powered educational systems.\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003EThrough a multi-year longitudinal co-design process with data scientists as well as several deployments and integrations into real-world educational settings, this thesis demonstrates how human-in-the-loop visual analytics can transform opaque LLM pipelines into transparent, iterative, and trustworthy development processes, ultimately supporting the responsible integration of LLMs into high-stakes learning environments. The outcomes of this thesis have been disseminated through multiple publications in top journals and conferences, advancing the state of the art in visual analytics, human-computer interaction, artificial intelligence, and educational technology by establishing new methods, systems, and design principles for making LLM behavior interpretable in pedagogical contexts.\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E","summary":"","format":"limited_html"}],"field_subtitle":"","field_summary":[{"value":"\u003Cp\u003EDetecting and Mitigating Pedagogical Risks in Large Language Models With Visual Analytics\u003C\/p\u003E","format":"limited_html"}],"field_summary_sentence":[{"value":"Detecting and Mitigating Pedagogical Risks in Large Language Models With Visual Analytics"}],"uid":"27707","created_gmt":"2026-04-06 17:51:48","changed_gmt":"2026-04-06 17:52:23","author":"Tatianna Richardson","boilerplate_text":"","field_publication":"","field_article_url":"","field_event_time":{"event_time_start":"2026-04-27T10:00:00-04:00","event_time_end":"2026-04-27T12:00:00-04:00","event_time_end_last":"2026-04-27T12:00:00-04:00","gmt_time_start":"2026-04-27 14:00:00","gmt_time_end":"2026-04-27 16:00:00","gmt_time_end_last":"2026-04-27 16:00:00","rrule":null,"timezone":"America\/New_York"},"location":"TSRB room 334 (VIS Lab) ","extras":[],"groups":[{"id":"221981","name":"Graduate Studies"}],"categories":[],"keywords":[{"id":"100811","name":"Phd Defense"}],"core_research_areas":[],"news_room_topics":[],"event_categories":[{"id":"1788","name":"Other\/Miscellaneous"}],"invited_audience":[{"id":"78771","name":"Public"}],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[],"email":[],"slides":[],"orientation":[],"userdata":""}}}