As the use of autonomous agents expands, agents are being tasked with completing more complex tasks in increasingly challenging environments. To complete these tasks agents must make decisions with limited information and onboard resources such as low computational power and inexpensive measurement units. One prevailing method to mitigate these challenges is to generate a solution, i.e., a reference trajectory, beforehand and then design a control law for the agents to track the a priori generated solution. Typically, an optimization algorithm uses a priori known data and dynamic models to generate the offline solution. However, in some environments it may not be possible to generate a solution offline. For example, we cannot generate offline optimal trajectories for agents to navigate an unmapped cave system in a search and rescue mission. One approach that has been proposed to enable agents to make decisions online is to use an optimization algorithm within the control loop. However, vital questions arise regarding stability, performance, and implementability when proposing to use optimization in-the-loop. Therefore, this dissertation addresses some of these challenges that arise when considering optimization in-the-loop onboard agents with limited information and onboard resources.