This thesis investigates how pretrained vision and language models can be effectively integrated into robotic manipulation systems to enhance generalization and scalability. While recent advances in data-driven robotic policies have enabled impressive performance across a wide range of tasks, they remain limited by the high cost of data collection, poor generalization to new environments, and the complexity of integrating perception, planning, and control into unified systems. In contrast, large vision and language models trained on internet-scale data exhibit strong generalization and reasoning capabilities, motivating their incorporation into robotics. To this end, this thesis presents a series of methods that leverage pretrained models across multiple levels of robotic functionality. We first propose a language-conditioned grasping framework that enables task-aware grasp selection through multimodal fusion of visual and linguistic inputs. Next, we introduce a keypoint-based representation for 6-DoF grasp generation, allowing effective transfer of pretrained 2D visual knowledge to 3D manipulation. For high-level reasoning, we develop a schema-guided multi-agent framework that utilizes large language models and scene graphs for spatial understanding and task planning. Finally, we analyze Vision-Language-Action models through the lens of visual conditioning, and propose a training framework that improves visual grounding and downstream task performance.