Human behavior modeling is a critical step to develop AI agents that are able to assist us in various tasks. In contrast with learning objects, scenes and textures, human behaviors are inherently purposeful, guided by underlying intentions and goals. Additionally, human behaviors involve precise and adaptive interactions with the environment, characterized by fine-grained and nuanced control. The two key differences require innovative approaches for AI models to understand our intentions in the behaviors, and capture the nuance of our actions in different tasks. In my thesis proposal, I will elaborate my research on leveraging multimodal inputs to capture the underlying intentions and enable precise controllability on human actions in both understanding and generation problems. My thesis includes four chapters: audio-visual gaze anticipation, multimodal social behavior understanding, text-guided egocentric action generation, and training-free text-image conditioned action generation. The ultimate goal of my research is to enable AI models to better understand and interact with people, paving the way towards human-centric artificial intelligence.