This research investigates approaches to mmWave beamforming practices to improve upon existing methods, in particular for wireless communication standards, such as 5G/New Radio (NR) and 802.11ad/ay. The current methods rely on an inefficient linear search over all possible steering directions for initial alignment, and require highly manual configurations to maintain alignment over time. We leverage state-of-the-art algorithms from areas such as reinforcement learning to improve upon these existing practices, in particular Multi-Armed Bandit (MAB) algorithms. Our work shows that our algorithms are competitive with scenarios in which full Channel State Information (CSI) is exploited for Bayesian decision making on the steered beam directions. Our algorithms also provide a low computational overhead alternative to algorithms considered optimal in sample complexity as a trade-off. We review the current state-of-the-art algorithms for initial alignment and tracking in Chapter 1. In Chapter 2, we describe our system model that precisely defines the problem space in which we make contributions. We also show a hierarchical codebook induces structure in our system model that allows opportunities for efficient exploitation. Then, we present Successive Subtree Elimination (SSE) in Chapter 3, an algorithm for the initial beam alignment problem that exploits the structure induced by a hierarchical beamforming codebook in a level-by-level approach. In Chapter 4, we provide an algorithm, Hierarchical Optimal Sampling for Unimodal Bandits (HOSUB), that uses depth-first search through the hierarchical graph to more quickly align than the level-by-level approach used in other algorithms using a hierarchical codebook. In Chapter 5, we extend SSE initial alignment algorithm to compensate for motion. Our extended algorithm, Dynamic Beam Zooming (DBZ), adjusts the beamforming vector selection to widen and narrow the beamwidth to mitigate the severe outages associated with mmWave channels. In Chapter 6, we demonstrate the trade-off between using optimal algorithms adapted for tracking versus other algorithms that require significantly less computational overhead. Lastly, in Chapter 7 we provide perspectives and conclusions on our major contributions.