The objective of the proposed research is to further scientific understanding of empirical phenomena in modern machine learning systems by examining how properties of the features, or data representations, used during training affect the statistical properties of overparameterized and high-dimensional models. We begin by showing that even in classical kernel methods, where the feature map is fixed, surprising optimization and generalization behaviors can emerge in sufficiently overparameterized settings for many natural families of features. Secondly, we design empirically and theoretically justified principles for evaluating the quality of pre-trained data representations from the perspective of generalization and robustness in classification tasks. Finally, we consider models such as neural networks that directly learn features from training data, and we study how and when such models can effectively adapt to low-dimensional structure without overfitting to noise. Overall, the proposed research aims to discover basic principles about data representations which provide theoretical support for common empirical observations, reveal sources of potential robustness failures, and inform the design of interpretable models and training schemes.