This thesis investigates how feature representations influence generalization and robustness in high-dimensional and overparameterized machine learning (ML) models. Using classical linear and kernel methods as a starting point, we first demonstrate that these simpler models can exhibit surprising optimization and generalization behaviors empirically observed in neural networks, such as benign overfitting and equivalence between different optimization objectives. We then develop new results for high-dimensional linear classification with imbalanced classes and use these results to develop useful heuristics for realistic pre-trained classification models. The second part of the thesis begins by challenging the use of linear and kernel methods as a framework for understanding modern ML models, showing that high-dimensional kernel methods suffer from fundamental approximation limitations not seen in neural networks. Finally,  we study a simple modification of the linear model that begins to bridge this gap, offering insight into how learnable feature maps can enable favorable generalization in high dimensions.  Overall, this dissertation aims to discover basic principles that provide theoretical support for common empirical observations, reveal sources of potential robustness failures, and inform the design of interpretable models and training procedures.