Modern datacenters form the foundation of today's digital infrastructure, supporting large-scale web services, enterprise cloud platforms, and emerging generative AI applications that process and exchange massive volumes of data. As processors continue to scale in core count and computational throughput, the disparity between compute capability and memory performance has become a critical bottleneck-manifesting as limitations in memory capacity, bandwidth, and cost. This growing imbalance, compounded by the slowdown of Moore's Law and increasing system complexity, poses a fundamental challenge to sustaining performance for data-intensive and highly parallel workloads. Addressing these challenges requires rethinking the memory hierarchy through innovations that jointly consider workload characteristics, hardware capabilities, and system-level interactions. This thesis presents a holistic, cross-layer co-design approach to overcome the memory wall by optimizing the memory hierarchy across chip, server, and cluster levels. At the chip level, HinTM enhances on-chip cache efficiency and transactional concurrency by mitigating capacity aborts in hardware transactional memory systems. At the server level, SURGE dynamically harvests idle I/O bandwidth over CXL links to boost effective memory bandwidth and reduce access latency under bandwidth-bound conditions. At the cluster level, COMET provides a composable modeling and co-optimization framework that balances compute, memory, and interconnect resources to accelerate distributed AI training. Together, these contributions advance the design of scalable, efficient, and workload-aware memory systems that sustain high performance across parallel and bandwidth-intensive computing environments.