Self Supervised Learning (SSL) has emerged as the dominant pre-training paradigm for virtually all modern foundation models. The general premise behind SSL is to pre-train a backbone (foundation) model on a large corpus of unlabeled data with the intent of forming robust representations that are transferable to potentially any downstream task.  Despite the empirical success of these methods, current SSL algorithms still face numerous generalization challenges. This includes specific issues such as poor transferability to data settings with localized fine-grained features, degradation of features over long training horizons, and reduced performance due to the presence of noise or other domain-specific data imbalances.  In this dissertation, we argue that these generalization challenges are largely a result of the suboptimal design of standard SSL optimization objectives. Current SSL algorithms utilize a fixed objective throughout training that does not adapt to the emergence of training dynamics unique to individual data settings. In general, very few works have analyzed the underlying training dynamics that influences the formation of SSL representations. Additionally, these works have not explored how to exploit these training dynamics in order to inform the design of SSL optimization algorithms. The goals of this dissertation are to 1) identify the components that reflect quality SSL representation spaces, 2) use these components to characterize SSL training dynamics in terms of distinct phases based on local and global relationships as well as info-theoretic representational dynamics, 3) utilize these insights to develop SSL algorithms that adapt to these identified dynamics in a dataset specific manner, and 4) demonstrate that our developed SSL algorithms result in state of the art performance across a wide range of application settings that include medicine, autonomous vehicle sensors, subsurface imaging, and large scale natural image datasets.