Date: Wednesday, November 18, 2026

Time: 1:00 PM to 3:00 PM Eastern Time (US)

Join in Person: KACB 1212

Join Virtually:  https://teams.microsoft.com/meet/2759987732877814?p=LV5TM5UdDfjS93xQrM

Tiansheng Huang

https://huangtiansheng.github.io/

CS Ph.D. Candidate

School of Computer Science

College of Computing

Georgia Institute of Technology

Committee:

Dr. Santosh Pande — Advisor and Committee Chair, School of Computer Science, Georgia Institute of Technology

Dr. Ling Liu— School of Computer Science, Georgia Institute of Technology

Dr. Calton Pu — School of Computer Science, Georgia Institute of Technology

Dr. Steve Mussmann — School of Computer Science, Georgia Institute of Technology

Dr. Ananth Balashankar — Google DeepMind

Abstract:

    Post-training has been extensively used by mainstream Large Language Model (LLM) service providers to customize the model for downstream applications. However, post-training methods (e.g., supervised fine-tuning, reinforcement learning) might be exploited by the malicious attackers to subvert the existing safety alignment of the model, causing serious safety concerns for the service provider. In this dissertation, we propose new post-training recipe to preserve the model's safety alignment against the above-mentioned harmful post-training attacks.

    In the first part, we study harmful fine-tuning attack, in which the attackers upload data with harmful responses and use them to subvert the model's safety alignment via supervised fine-tuning (SFT). For mitigation, we propose defenses that are applied i) before attack, ii) during attack and iii) after attack.   For the first line of work, we propose a series of defense methods: Vaccine, Booster, Tcell, which impose stronger safety alignment on the model to increase the model's robustness before an attack happens.  To reach the goal, Vaccine utilizes a min-max optimization to make the model robust to worst-case perturbations, Booster simulates harmful perturbations to flatten the harmful loss landscape, and Tcell further aligns the direction of safety gradient and the harmful gradient.  Next, we propose a fine-tuning stage defense that is applied during attack, dubbed Lisa. Lisa modifies the post-training recipe for downstream fine-tuning via alternating between the safety-alignment state and the user fine-tuning state, and it introduces a proximal term to constrain the drift of each state to ensure convergence stability. This alternating process reinforces the safety capability of the model while learning customized capability during downstream fine-tuning from users. Finally, we propose a post-fine-tuning stage defense dubbed Antidote, which aims to recover the model's safety alignment after an attack via pruning out harmful parameters. The three lines of post-training recipes can be combined to provide better defense due to their different timing of application within the model post-training lifecycle.      

   The second part of our research explores harmful agentic reinforcement learning, in which the attackers use a harmful reward function to gradually subvert the model's safety alignment during agentic reinforcement learning (RL). For mitigation, we propose a defense dubbed Bait that applies during the RL process. The core principle of Bait is to construct a reliable detection criterion for deciding whether a harmful reward function or a benign reward function is applied. If the detection is positive, i.e., the reward function is classified as a harmful reward function, the service provider reverses the reward value to apply a penalty during the policy optimization to suppress harmful learning.  

   Finally, in the last part,  we outline future research directions of harmful post-training. Particularly, we want to

   emphasize the emerging risk of Recursive Self-Improvement (RSI), a post-training recipe that allows a model to self-evolve without human supervision. RSI led by an LLM itself might subvert the human-imposed safety alignment, causing a great risk of loss of control.