The objective of the proposed research is to develop language-model-based methods to address the problem of situated and multimodal interactive conversations in scientific papers. While significant progress has been made in large multimodal language models capable of multi-turn dialogue, creating a conversational assistant that can effectively navigate and discuss structured knowledge sources remains an open challenge, particularly within the domain of scientific literature. Scientific documents present a unique set of challenges due to their inherently multimodal nature. Researchers utilize various formats to convey complex information: figures illustrate key concepts, equations express mathematical principles, and tables summarize experimental setups and results. Moreover, the interconnected nature of these elements—with researchers frequently cross-referencing figures, tables, and equations throughout the text—adds another layer of complexity. This research seeks to address these challenges by developing methods that can seamlessly integrate and interpret these diverse modalities, enabling more sophisticated and context-aware interactions with scientific content.