Date: Thursday, October 10, 2024

Time: 9 a.m. - 11 a.m. ET (US)

Location: Technology Square Research Building (TSRB) 334

Virtual meeting (hybrid): Click here to join Zoom meeting

Alexander Bendeck

Ph.D. Student in Computer Science 

School of Interactive Computing 

Georgia Institute of Technology 

Committee

Dr. John Stasko (Advisor) - School of Interactive Computing, Georgia Institute of Technology

Dr. Alex Endert - School of Interactive Computing, Georgia Institute of Technology

Dr. Clio Andris - School of City and Regional Planning, Georgia Institute of Technology

Dr. Cindy Xiong Bearfield - School of Interactive Computing, Georgia Institute of Technology

Dr. Ross Maciejewski - School of Computing and Augmented Intelligence, Arizona State University

Abstract

Advances in generative artificial intelligence have led to the development of pre-trained large language models (LLMs) which are widely available and broadly useful. For data visualization researchers, LLMs have the promise to extend existing research threads in exciting directions, potentially super-charging visualization systems with their vast domain knowledge and computational power. However, LLM-powered systems pose new challenges for both visualization researchers and our intended system users. For instance, well-documented hallucination and inconsistency issues with LLMs can inhibit visualization system performance and erode user trust. We also have little formal understanding of LLMs’ ability to help data analysts with specific tasks.

The aim of my thesis work is to study the potential use of LLMs as “virtual domain experts” during visual data analysis. This includes two main goals: First, to evaluate LLMs at applying their knowledge bases to data- and chart-centric tasks; and second, to study user task completion, satisfaction, and trust for LLM-powered visualization systems. I addressed the first goal through an empirical evaluation of the GPT-4V multimodal language model on a suite of visualization literacy tasks, demonstrating the state of the art in LLM performance at reading and understanding visualizations. I propose subsequent work to address both goals by assessing LLMs’ domain knowledge and generative capabilities on two specific tasks: question answering and data integration. For each task, I will conduct formative studies, empirical evaluations, and design probes using prototype visualization systems, exploring both technical and human-centered perspectives on the use of LLMs during visual data analysis.