The objective of the proposed research is to study the impact of estimating and integrating components of the underlying data distribution into the process of representation learning. Representation learning refers to creating abstractions of data that allow extraction of useful information for a target downstream task. Effective representations capture the posterior distribution of the underlying explanatory factors for the observed input. Explanatory factors are any component of the data distribution that causes variation between samples. Low level factors include the presence of noise and various types of data augmentations while higher level factors are indicative of domain-specific semantic relationships. Ideally, representations should reflect multiple explanatory factors as well as a hierarchical organization of factors. One state of the art representation learning strategy known as contrastive learning involves producing embeddings of similar data points (positives) that are projected closer to each other than that of dissimilar data points (negatives). Traditional approaches do this by performing data augmentation on the sample of interest (anchor point) to generate a positive pair and then treating all other instances in the batch as the negative set. The main issue with this approach is that they do not enforce representations that reflect the structure of explanatory factors within an application domain. For example, the medical domain has distributions that exist as a result of relationships in terms of clinical data and disease severity, fisheye sensor data has distortion distributions that arise from noisy acquisition conditions, and natural image data has distributions that arise from taxonomy relationships between classes. In this dissertation, I propose to estimate the structure of component distributions and integrate them into contrastive learning frameworks. I propose to show the efficacy of this approach from a domain-specific and generalized point of views.