Autonomous networked systems are increasingly tasked with completing complex assignments in challenging environments. In these settings, individual agents must remain safe while, as a collective, making progress towards completing the network's objective. It is especially important, as these systems are brought into the public sphere, that preserving the safety of humans is kept as a top priority. Autonomous systems typically make decisions using online optimization so that their control actions are determined by minimizing a given objective function. One technique to develop safe autonomous behavior is to incorporate a constraint into optimization problems that, when satisfied, guarantees that the system remains in a safe set of the state space. One challenge with this approach is that it is unknown beforehand if this problem will be continuously feasible, that is, whether it will always be possible to satisfy the constraint that guarantees safe behavior. This dissertation addresses this challenge by formulating a priori techniques that guarantee continued system safety prior to implementation. In the latter half of this dissertation, we address the challenge of reliable network performance, that is, the ability of a group of autonomous agents to collaboratively complete an assigned objective. Agents within the network make decisions using their own locally computed information and information gathered from communication with other agents in the network. As a result of communication delays or varying computational power, agents often make decisions using outdated information. In this dissertation, we propose a distributed algorithm that is guaranteed to accomplish its objective even with arbitrarily outdated information. We then analyze a collection of algorithms that can tolerate this type of asynchrony, and we derive formal assurances on which algorithm is guaranteed to accomplish its objective in the smallest number of iterations. We also consider the setting in which the level of asynchrony within the network is bounded. In such a setting, we propose a general class of algorithms and show that it is reliably convergent.