The research on robotic systems capable of assisting humans in common daily activities spans a diverse array of interdisciplinary domains, each contributing essential components to the overall functionality and intelligence of the system. To coexist and assist humans in the same space, robots are required to process and understand natural language, interpret neighboring scenes, plan sequential actions, navigate through the environment, and manipulate objects. Natural language understanding and perceptual scene understanding are responsible for grounding multi-modal inputs from the external environment, while navigation and manipulation are responsible for performing actions to physically interact with the environment. Central to integrating sensing, understanding, and interaction with the environment is task and motion planning (TAMP). Deep learning has demonstrated remarkable efficacy in processing highly complex and variable data, such as perception and language, which significantly affects the design of robotic systems. This thesis focuses on enhancing specification-based representation learning for TAMP in robotic systems, designed to assist humans in daily activities. These representations serve as a conceptual bridge, harnessing the feature-learning capability of deep learning while seamlessly integrating with classical planning schemes. We aim to investigate representations that align with task-related and motion-related specifications, facilitating the development and practical deployment of robots in real-world settings.