Sparse neural networks (Sparse NNs) are characterized by having fewer connections between consecutive layers compared to traditional fully connected, or dense NNs. Historically, sparsity has been studied post-training to enhance inference efficiency and as a regularization mechanism to improve generalization. However, additional benefits beyond these areas remain underexplored. In this thesis, we investigate sparse NNs, various sparsity patterns, and their broader benefits. First, we introduce PHEW (Path with Higher Edge-Weights), a novel method for constructing sparse NNs at initialization, without using any training data. PHEW does not make any task-specific assumptions; instead, it exploits structural properties inherent in dense NNs that promote faster convergence and better generalization, thereby reducing the computational burden of training dense NNs. Second, we propose Neural Sculpting, a technique to uncover the underlying hierarchically modular task structure within NNs. Neural Sculpting uses an iterative process of pruning both units and edges during training, followed by network analysis to detect functional modules and infer hierarchical relationships between them. This method enhances the interpretability of NNs by guiding them to reflect the task’s inherent structure. Finally, we leverage structural information about the task’s hierarchical modularity to enhance NN performance by aligning the architecture at initialization with the task’s structure. Specifically, we compare architectures ranging from monolithic dense NNs, which assume no prior knowledge, to hierarchically modular NNs with shared modules, which leverage sparsity, modularity, and module reusability. Incorporating modularity and module reuse significantly enhances learning efficiency and generalization, particularly in data-scarce scenarios. In conclusion, this thesis demonstrates that sparse NNs offer not only enhanced training and inference efficiency but also superior interpretability and generalization capabilities.