Date: Friday, July 17th 2026
Time: 11:00 - 12:30 PM EST
Location: Coda C1115 Druid Hills
Zoom: https://gatech.zoom.us/j/2123115504?pwd=eTJFZWR1dXN6ZlZ3WGtwRlMzQmZNQT09
Yao Dou
Ph.D. Student
School of Interactive Computing
Georgia Institute of Technology
Committee members
Dr. Wei Xu (advisor): School of Interactive Computing, Georgia Institute of Technology
Dr. Alan Ritter: School of Interactive Computing, Georgia Institute of Technology
Dr. Polo Chau: School of CSE, Georgia Institute of Technology
Dr. Michel Galley: Senior Principal Research Manager at Microsoft Research
Dr. Dipanjan Das: Senior Director of Research at Google Deepmind
Abstract:
As large language models (LLMs) move into real-world, open-ended applications, evaluating them becomes both more important and more difficult. My thesis develops evaluation methods that go beyond multiple-choice accuracy to handle multi-turn interaction, long-context tasks, and fine-grained text quality.
In this thesis defense, I will first present our work on evaluating multi-turn assistants with user simulators. I introduce SimulatorArena, a benchmark of annotated human--LLM conversations that assesses how closely simulators match real user behavior and how well their assistant evaluations align with human judgments, and show that simulators conditioned on structured user profiles align closely with human ratings, making scalable multi-turn evaluation practical. I will next introduce GAVEL, an evaluation framework for long-context legal summarization, where case documents often exceed 100K tokens and expert summaries are long and information-dense. GAVEL-Ref evaluates model summaries against expert-written references through checklist, residual-fact, and writing-style evaluation, and GAVEL-Agent navigates the case documents with tools to extract checklist items directly, achieving competitive performance with substantially lower token usage and generalizing to the medical domain.
]]>