Dear faculty members and fellow students,

You are cordially invited to my PhD thesis defense on Monday, November 23rd.

Title: Adaptive and Robust Model Fusion for Multi-Modal Learning Systems

Date: Monday, November 23, 2026

Time: 1:00 PM - 3:00 PM Eastern Time (US)

Location: KACB 3402

Virtual (Zoom): https://gatech.zoom.us/j/94537626884?pwd=f2aDUdkbH2u8p6w9oLP0zMBdv6CJCN.1

Selim Furkan Tekin
Ph.D. Candidate in Computer Science
School of Computer Science
Georgia Institute of Technology

Committee members:

1. Dr. Ling Liu (Advisor)  - School of Computer Science, Georgia Institute of Technology
2. Dr. Francisco Romero - School of Computer Science, Georgia Institute of Technology
3. Dr. Anand Padmanabha Iyer - School of Computer Science, Georgia Institute of Technology
4. Dr. Calton Pu - School of Computer Science, Georgia Institute of Technology
5. Dr. Margaret Loper - Information and Communications Laboratory, Georgia Tech Research Institute
6. Dr. Lakshmish Ramaswamy - School of Computing, University of Georgia

Abstract

The advancement of large-scale AI models in recent years has shown their remarkable performance across a broad spectrum of zero- and one-shot tasks across language and vision modalities. However, based on the benchmark studies and model release announcements revealed that even the newest released or largest models tend to carry some weaknesses despite the unique strengths, indicating no one model can be the persistent winner in generalization performance across all tasks. Complementing this, user-generated queries tend to have varying complexity and involve simple common knowledge tasks or complex reasoning or specific domain knowledge. For simple queries involving superficial understanding, smaller models can do a perfect job at a minimal cost. Hence, there is a challenge in adequately choosing a model or group of models based on the client-user constraints and the downstream learning tasks at hand. This dissertation is dedicated to tackling the above grand challenges by designing and developing innovative methodologies, algorithms, and optimizations in selecting, refining, and integrating multiple gen-AI models. 

Our research began with a comprehensive empirical study, which led to three observations: First, in classical few-shot learning methods, ensemble gains come largely from two channels: structural (model architecture) and geometric (distance metric) diversity–motivating heterogeneous model selection. We later found that the same holds for modern few-shot learners such as LLMs and VLMs. Second, larger heterogeneous ensemble teams may not necessarily offer better generalization performance, because some members make redundant predictions that add inference cost without adding new information. Third, the language models and vision models require different multi-stage representation learning to effectively capture duplicated learning capabilities. Motivated by these observations, we have made four novel and fundamental contributions:

First, we have developed a novel diversity computation framework that can be tailored for both language and vision representation learning to capture the disagreement correlation among multiple gen-AI models (LLMs, LVMs, and VLMs). 

Second, we have designed the first of its kind diversity-optimized ensemble pruning framework, which supports LLM and VLM ensemble pruning. The pruning utilizes genetic search optimization on introduced metrics: the focal-diversity metric and focal Central Kernel Alignment (CKA) to identify informative complementary representations for vision-language ensemble pruning. 

Third, we study learning-to-combine architectures and develop two diversity-optimized ensemble fusion methods. The first is a supervised approach that uses diversity-driven pruning to select an ensemble team offline and then learns to fuse its outputs, achieving stronger generalization than state-of-the-art methods. The second selects the team and fuses outputs dynamically at inference time, using a two-stage reinforcement learning framework.

Finally, we address model routing and fusion through joint optimization, learning how to select, reason, and refine across multiple VLMs. We develop two routing systems trained with reinforcement learning. The first starts from prior beliefs about each VLM’s performance: a routing policy chooses which model to consult, and a fusion policy decides whether to keep consulting or answer. The second targets physical reasoning in real-world scenes: it matches the skills a query requires to each VLM’s perception strengths, routes to models in sequence, and refines a lightweight policy using critiques from a tool-augmented critic that extracts visual clues. In both lines of work, we compare our approaches against widely used AI chatbots and show that our approaches succeed in real-world scenarios where existing models often fail. 

Collectively, these contributions offer new ways to build teams of diverse AI models that reason together, combine their outputs, and verify their answers. Such methods become increasingly important as the number and variety of large-scale models continue to grow.