The objective of the proposed research is to achieve memory and storage efficient machine learning by leveraging system-level optimizations and developing next-generation high-level synthesis (HLS) tools for agile architecture design. At the system level, this research focuses on reducing communication overhead among edge devices and minimizing data storage requirements by proposing Rapid-INR and Residual-INR. Those two works employ implicit neural representations (INRs) to achieve efficient data compression and facilitate accelerated training and communication-efficient on-device learning. To address the suboptimal memory and storage usage across multi-level hierarchies, this research also develops an intelligent caching mechanism called ICGMM for CXL-enabled memory expansion, together with a cost-effective multi-node training system COMETS using memory pooling and sharing. At the tool level, faced with the challenge that current HLS tools are suitable for fixed dataflow specific accelerator designs but not general architecture designs with flexible dataflows and workload mapping, in the future, this research proposes to develop a next-generation architectural-oriented HLS tool called ArchHLS. ArchHLS allows flexible architecture extraction and definition, decoupling of architecture design and computation (workload compilation and mapping), and automatic architecture evolution to new algorithms.