{"684314":{"#nid":"684314","#data":{"type":"news","title":"Georgia Tech\u2019s Jill Watson Outperforms ChatGPT in Real Classrooms","body":[{"value":"\u003Cp\u003EA new version of Georgia Tech\u2019s virtual teaching assistant, Jill Watson, has demonstrated that artificial intelligence can significantly improve the online classroom experience. Developed by the \u003Ca href=\u0022https:\/\/dilab.gatech.edu\/\u0022\u003EDesign Intelligence Laboratory\u003C\/a\u003E (DILab) and the \u003Ca href=\u0022https:\/\/aialoe.org\u0022\u003EU.S. National Science Foundation AI Institute for Adult Learning and Online Education\u003C\/a\u003E (AI-ALOE), the latest version of Jill Watson integrates \u003Ca href=\u0022https:\/\/openai.com\/\u0022\u003EOpenAI\u003C\/a\u003E\u2019s ChatGPT and is outperforming OpenAI\u2019s own assistant in real-world educational settings.\u003C\/p\u003E\u003Cp\u003EJill Watson not only answers student questions with high accuracy. It also improves teaching presence and correlates with better academic performance. Researchers believe this is the first documented instance of a chatbot enhancing teaching presence in online learning for adult students.\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EHow Jill Watson Shaped Intelligent Teaching Assistants\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EFirst introduced in 2016 using IBM\u2019s Watson platform, Jill Watson was the first AI-powered teaching assistant deployed in real classes. It began by responding to student questions on discussion forums like Piazza using course syllabi and a curated knowledge base of past Q\u0026amp;As. Widely covered by major media outlets including \u003Cem\u003EThe Chronicle of Higher Education\u003C\/em\u003E, \u003Cem\u003EThe Wall Street Journal\u003C\/em\u003E, and \u003Cem\u003EThe New York Times\u003C\/em\u003E, the original Jill pioneered new territory in AI-supported learning.\u003C\/p\u003E\u003Cp\u003ESubsequent iterations addressed early biases in the training data and transitioned to more flexible platforms like Google\u2019s BERT in 2019, allowing Jill to work across learning management systems such as EdStem and Canvas. With the rise of generative AI, the latest version now uses ChatGPT to engage in extended, context-rich dialogue with students using information drawn directly from courseware, textbooks, video transcripts, and more.\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EFuture of Personalized, AI-Powered Learning\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EDesigned around the Community of Inquiry (CoI) framework, Jill Watson aims to enhance \u201cteaching presence,\u201d one of three key factors in effective online learning, alongside cognitive and social presence. Teaching presence includes both the design of course materials and facilitation of instruction. Jill supports this by providing accurate, personalized answers while reinforcing the structure and goals of the course.\u003C\/p\u003E\u003Cp\u003EThe system architecture includes a preprocessed knowledge base, a MongoDB-powered memory for storing conversation history, and a pipeline that classifies questions, retrieves contextually relevant content, and moderates responses. Jill is built to avoid generating harmful content and only responds when sufficient verified course material is available.\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EField-Tested in Georgia and Beyond\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EThe first AI-powered teaching assistant was developed for Georgia Tech\u2019s Online Master of Science in Computer Science (OMSCS) program. By fall 2023, Jill Watson was deployed in Georgia Tech\u2019s OMSCS artificial intelligence course, serving more than 600 students, as well as in an English course at Wiregrass Georgia Technical College, part of the Technical College System of Georgia (TCSG).\u003C\/p\u003E\u003Cp\u003EA controlled A\/B experiment in the OMSCS course allowed researchers to compare outcomes between students with and without access to Jill Watson, even though all students could use ChatGPT. The findings are striking:\u003C\/p\u003E\u003Cul\u003E\u003Cli\u003EJill Watson\u2019s accuracy on synthetic test sets ranged from 75% to 97%, depending on the content source. It consistently outperformed OpenAI\u2019s Assistant, which scored around 30%.\u003C\/li\u003E\u003Cli\u003EStudents with access to Jill Watson showed stronger perceptions of teaching presence, particularly in course design and organization, as well as higher social presence.\u003C\/li\u003E\u003Cli\u003EAcademic performance also improved slightly: students with Jill saw more A grades (66% vs. 62%) and fewer C grades (3% vs. 7%).\u003C\/li\u003E\u003C\/ul\u003E\u003Cp\u003E\u003Cstrong\u003EA Smarter, Safer Chatbot\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EWhile Jill Watson uses ChatGPT for natural language generation, it restricts outputs to validated course material and verifies each response using textual entailment. According to a study by Taneja et al. (2024), Jill not only delivers more accurate answers than OpenAI\u2019s Assistant but also avoids producing confusing or harmful content at significantly lower rates.\u003C\/p\u003E\u003Cp\u003ECompared to OpenAI\u2019s Assistant, Jill Watson (ChatGPT) not only achieves higher accuracy but also produces confusing or harmful content at significantly lower rates. Jill Watson answers correctly 78.7% of the time, with only 2.7% of its errors categorized as harmful and 54.0% as confusing. In contrast, OpenAI\u2019s Assistant demonstrates a much lower accuracy of 30.7%, with harmful failures occurring 14.4% of the time and confusing failures rising to 69.2%. Additionally, Jill Watson has a lower retrieval failure rate of 43.2%, compared to 68.3% for the OpenAI Assistant.\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EWhat\u2019s Next for Jill\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EThe team plans to expand testing across introductory computing courses at Georgia Tech and technical colleges. They also aim to explore Jill Watson\u2019s potential to improve cognitive presence, particularly critical thinking and concept application. Although quantitative results for cognitive presence are still inconclusive, anecdotal feedback from students has been positive. One OMSCS student wrote:\u003C\/p\u003E\u003Cp\u003E\u003Cem\u003E\u201cThe Jill Watson upgrade is a leap forward. With persistent prompting I managed to coax it from explicit knowledge to tacit knowledge. Kudos to the team!\u201d\u003C\/em\u003E\u003C\/p\u003E\u003Cp\u003EThe researchers also expect Jill to reduce instructional workload by handling routine questions and enabling more focus on complex student needs.\u003C\/p\u003E\u003Cp\u003EAdditionally, AI-ALOE is collaborating with the publishing company John Wiley \u0026amp; Sons, Inc., to develop a Jill Watson virtual teaching assistant for one of their courses, with the instructor and university chosen by Wiley. If successful, this initiative could potentially scale to hundreds or even thousands of classes across the country and around the world, transforming the way students interact with course content and receive support.\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EA Georgia Tech-Led Collaboration\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003EThe Jill Watson project is supported by Georgia Tech, the US National Science Foundation\u2019s AI-ALOE Institute (Grants #2112523 and #2247790), and the Bill \u0026amp; Melinda Gates Foundation.\u003C\/p\u003E\u003Cp\u003ECore team members are Saptrishi Basu, Jihou Chen, Jake Finnegan, Isaac Lo, JunSoo Park, Ahamad Shapiro and Karan Taneja, under the direction of professor Ashok Goel and Sandeep Kakar. The team works under Beyond Question LLC, an AI-based educational technology startup.\u003C\/p\u003E","summary":"","format":"limited_html"}],"field_subtitle":"","field_summary":[{"value":"\u003Cp\u003EGeorgia Tech\u2019s latest version of Jill Watson, a virtual teaching assistant, is showing how artificial intelligence can improve online learning. Developed by the Design Intelligence Laboratory and the NSF AI Institute for Adult Learning and Online Education, the system now integrates OpenAI\u2019s ChatGPT but outperforms OpenAI\u2019s own assistant in accuracy, safety, and educational impact.\u003C\/p\u003E","format":"limited_html"}],"field_summary_sentence":[{"value":"Georgia Tech\u2019s Jill Watson, now powered by ChatGPT, is the first documented chatbot to enhance teaching presence in online learning, outperforming OpenAI\u2019s own assistant in accuracy, safety, and student outcomes."}],"uid":"36348","created_gmt":"2025-09-02 13:45:29","changed_gmt":"2025-09-09 13:24:53","author":"Breon Martin","boilerplate_text":"","field_publication":"","field_article_url":"","location":"Atlanta, GA","dateline":{"date":"2025-09-02T00:00:00-04:00","iso_date":"2025-09-02T00:00:00-04:00","tz":"America\/New_York"},"extras":[],"hg_media":{"677873":{"id":"677873","type":"image","title":"Georgia-Tech-s-Jill-Watson-Outperforms-ChatGPT-in-Real-Classrooms.png","body":null,"created":"1756820747","gmt_created":"2025-09-02 13:45:47","changed":"1756820747","gmt_changed":"2025-09-02 13:45:47","alt":"Georgia Tech\u2019s Jill Watson Outperforms ChatGPT in Real Classrooms","file":{"fid":"261822","name":"Georgia-Tech-s-Jill-Watson-Outperforms-ChatGPT-in-Real-Classrooms.png","image_path":"\/sites\/default\/files\/2025\/09\/02\/Georgia-Tech-s-Jill-Watson-Outperforms-ChatGPT-in-Real-Classrooms.png","image_full_path":"http:\/\/hg.gatech.edu\/\/sites\/default\/files\/2025\/09\/02\/Georgia-Tech-s-Jill-Watson-Outperforms-ChatGPT-in-Real-Classrooms.png","mime":"image\/png","size":3317148,"path_740":"http:\/\/hg.gatech.edu\/sites\/default\/files\/styles\/740xx_scale\/public\/2025\/09\/02\/Georgia-Tech-s-Jill-Watson-Outperforms-ChatGPT-in-Real-Classrooms.png?itok=DhRS832z"}}},"media_ids":["677873"],"groups":[{"id":"1214","name":"News Room"},{"id":"1188","name":"Research Horizons"},{"id":"660368","name":"Tech AI (Artificial Intelligence)"}],"categories":[],"keywords":[{"id":"192863","name":"go-ai"},{"id":"187915","name":"go-researchnews"}],"core_research_areas":[{"id":"193655","name":"Artificial Intelligence at Georgia Tech"}],"news_room_topics":[],"event_categories":[],"invited_audience":[],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[{"value":"\u003Cp\u003EBreon Martin\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E","format":"limited_html"}],"email":["breon@gatech.edu"],"slides":[],"orientation":[],"userdata":""}}}