The dissertation presents hardware and algorithm co design strategies to improve energy efficiency and reduce latency across different domain specific machine learning systems. It explores three distinct stages of the machine learning system stack: the sensor to model interface, the neural network architecture, and the hardware accelerator. The research demonstrates how optimizing algorithms and hardware together at each stage leads to meaningful performance gains tailored to specific applications. First, we develop energy efficient methods for the sensor to model interface using compute in memory accelerators and specialized quantization aware training based low precision models. Second, we design lightweight neural network architectures and adaptive sensing strategies for energy efficient and low latency radar applications. Third, we create low latency systems that combine distributed online learning methods with custom hardware acceleration to solve complex combinatorial optimization problems.