Learning-based techniques have become increasingly popular as a tool for improving autonomy of aerial robotics platforms. These methods typically lack safety guarantees using formal verification. We contribute to autonomy for aerial robotics by using learning-based techniques and a safe verification method for supervising unverified controllers. Specifically, we propose a navigation framework for urban air mobility and aerial robotics that uses real-time motion planning with continuous-time learning, adapting an actor-critic model predictive control approach for urban air mobility, and using reachability with control barrier functions for verifying these learning-based methods. Leveraging mixed monotonicity to quickly overapproximate reachable sets for aerial drone dynamics in real time. Additionally, we explore multi-agent reinforcement learning to teach drone swarms to cooperate in a multi-agent pursuit evasion problem. Finally, we contribute software tools for the aerial robotics community for simulating hardware code for the Crazyflie nano quadrotor.