This research develops non-collaborative spectrum sharing techniques that may be used by radars against dynamic and reactive in-band emitters. Historically, each user of RF spectrum was allocated a different frequency band, allowing for interference-free operation but inefficient use of spectrum resources, because of temporally sparse use in some allocated frequency bands. The proliferation of commercial cellular services in the mid-2000s strained spectrum resources, resulting in US government-led spectrum auctions, which changed spectrum allocation to a sharing model to more efficiently use limited spectrum resources. Spectrum sharing is a well-established field of research with a bias toward collaborative techniques intended for use by communication systems. Modern radar systems have distinct requirements and capabilities to make them candidates for non-collaborative spectrum sharing. Radars can rapidly change transmission frequency, location, direction, time, polarization, and waveform code to increase the efficiency of spectrum use while simultaneously minimizing interference. Existing non-collaborative spectrum sharing techniques use reactive sense and avoid, predictive models, and online learning. Importantly, prior non-collaborative spectrum sharing research evaluated spectrum sharing performance in the presence of time-varying emitters and has been limited in its examination of the utility of online learning. This work extends prior non-collaborative spectrum sharing by assuming that time-varying emitters react to other in-band spectrum users. Discrete and continuous Markov decision process (MDP) models are developed for radar spectrum sharing with appropriate reinforcement learning algorithms to control radar waveform selection. Radar spectrum sharing performance is assessed through simulation using the Q-Learning algorithm to make spectrum access decisions in the discrete MDP model and a linear function approximation algorithm using the deterministic policy gradient theorem is used in the continuous MDP model. Simulations for both MDP models include a variety of time-varying and reactive emitters and alternative radar spectrum access strategies for comparison.