Multi-agent systems are present in a wide range of domains, from physical systems (e.g., molecules, planets) and biological systems (e.g., host-pathogen interactions, neurons) to social systems (e.g., covid-19 spread, games with human players). While these systems have significant real-world applications, mathematically modeling their often-unknown dynamics is challenging. Deep learning offers a data-driven approach to modeling these systems without requiring extensive domain knowledge. However, collecting sufficient training data is difficult, as these systems evolve over time, and we may not even detect when the underlying dynamics change. Moreover, multi-agent systems are often driven by a large number of agents, making learning and prediction computationally expensive and inefficient. This research addresses these challenges by developing innovative algorithms and neural network architectures that can efficiently learn representations of the spatial arrangement of agents, forecast their trajectories and state transitions, and uncover hidden interaction graphs in unstructured and structured multi-agent systems, considering data and computation constraints.