In the past few years, a burgeoning field of research has emerged within the broader AI community known as "Embodied AI." This field encompasses various challenges, including the development of scene datasets and simulators used to train AI agents in diverse tasks, necessitating a comprehensive set of skills. Generally speaking, Embodied AI research projects assume a similar setup where an agent equipped with a set of sensors (usually an RGB and Depth camera) is trained to accomplish certain tasks such as navigation, pick-and-place, question answering etc... While there are many approaches and designs of such Embodied AI agents, this thesis focuses on methods involving intermediate scene representations. In contrast to end-to-end approaches, AI systems with intermediate explicit scene representations comprise two distinct modules: one for raw input sensor processing and a second one for planning and acting. Agents utilizing scene representations offer several advantages, including reduced susceptibility to the forgetting effect observed in their RNN counterparts, easier incorporation of inductive bias directly into the representations (e.g., geometrical constraints), and improved interpretability. Towards this end, this thesis serves as an investigation towards the design of such scene representations. We specifically research how to leverage 3D mapping techniques in order to build rich, dense and useful representations for Embodied AI applications. We start by studying representations in the form of 2D topdown metric maps. These 2D maps can store features, labels or geometrical information to form a useful training signal for the tasks of semantic mapping, navigation or question answering. We then study 3D representations in the forms of implicit maps applied for SLAM and 3D object-based maps for multi-object re-identification. Next we extend our research to dynamic scenes and explore 4D representations in the form of localized keyframes. Finally, the thesis also explores connections between these different representations while highlighting their strengths and limitations.