HVAC systems are essential for regulating indoor temperature, humidity, and air quality, making them significant energy consumers in U.S. buildings, accounting for nearly 40% of the total energy use. This dissertation focuses on designing human-in-the-loop energy-efficient HVAC controllers to reduce energy consumption while maintaining occupant comfort. The nonlinear thermal dynamics of buildings make traditional control methods ineffective. Reinforcement Learning (RL) is utilized to develop HVAC controllers in such complex environments. The proposed method combines model-based RL with model predictive control to avoid compounding errors of the existing imitation learning methods, demonstrating superior energy efficiency and indoor temperature regulation in EnergyPlus environment simulations. Occupants' varied thermal preferences pose a challenge for shared HVAC controllers. To address this, a meta-learning algorithm is used to develop a personalized thermal comfort model with minimal feedback instances. This method improves thermal comfort predictions with just five personalized thermal sensation votes, as validated by using the ASHRAE global thermal comfort database II. Based on the meta-learning method, we further present an algorithm to identify the best personalized thermal conditions for occupants. A thermal sensation generation model based on the PMV model further demonstrates the effectiveness of this identification algorithm. In addition, this dissertation leverages offline datasets and personalized thermal comfort models to derive a human-in-the-loop energy-efficient HVAC controller. Training with offline data reduces the risk of unsafe states during online RL training. An ensemble neural network is employed to predict thermal states, indicating data-covered regions. Combining the personalized thermal preference model and model-based offline reinforcement learning, the derived HVAC controller can adapt to different occupants' preferences with minimal personalized thermal feedback, ensuring safer training and practical deployment. Simulations confirm the proposed algorithm's effectiveness, maintaining personalized comfort with only a 1.91% increase in power consumption.