Deep learning systems have exhibited tremendous capabilities in decision-making tasks by learning neural representations. To achieve performance, these representations are often expected to implicitly learn invariances from large-scale training datasets, but often fail to generalize task-relevant and task-irrelevant features of data. As an alternative to implicitly learning this structure, the manifold hypothesis suggests that such representations should parameterize task-relevant features of each category with a few degrees of freedom, while separating representations of different categories. Motivated by this hypothesis, we propose techniques to incorporate a generative manifold model into neural representations by learning a dictionary of Lie group operators in the latent space of a deep neural network. We first discuss training techniques to learn this dictionary in an unsupervised manner, allowing for sampling, interpolation, and extrapolation along the manifold. We then discuss approaches in variational sparse coding, which dramatically increase the computational efficiency of the model in train and test time. Finally, we propose a contrastive learning algorithm which incorporates manifold feature augmentations to increase label efficiency. To make this possible, we learn local manifold statistics, allowing for sampled augmentations which preserve identity for a given input data point.