In this research, we propose a novel heterogeneous recurrent spiking neural network (HRSNN) that utilizes spike-timing-dependent plasticity (STDP) rules with varying neuronal and synaptic dynamics. We aim to model, characterize, and optimize this unsupervised learning method analytically and empirically. The thesis illustrates how the temporal event processing in such an HRSNN model offers significant advantages over traditional homogeneous RSNN (MRSNN), backpropagated RSNN (BPRSNN), and deep neural networks. By harnessing the diversity in the neuronal and synaptic timescales, we engineer more robust, generalizable models, which are otherwise not possible using supervised learning. Furthermore, STDP allows these learning models to train with considerably less data, learn the underlying dynamics in much fewer timesteps, and showcase remarkable noise resilience. However, one key drawback of using heterogeneity in the parameters is that the model becomes exponentially more complex and, thus, becomes much more challenging to train, and finding the optimal hyperparameters becomes exponentially more difficult as the number of neurons increases. Using standard activity-based pruning methods for HRSNN models makes the model extremely unstable and, thus, hard to train. Moreover, such pruning methods do not fully exploit the diversity in the neuronal timescales present in an HRSNN model. Thus, we introduce a novel Lyapunov Noise Pruning (LNP) algorithm that uses graph sparsification methods and utilizes Lyapunov exponents to design a stable sparse HRSNN from an untrained HRSNN model by leveraging the diversity in neuronal timescales in HRSNN. Going forward, we aim to exploit the rich dynamics of HRSNNs to tackle problems that have hitherto posed significant challenges for conventional DNN models. In particular, we intend to look into NP-hard combinatorial optimization and variational inequality problems. In harnessing the capabilities of HRSNNs, we have unveiled a pathway to not only more efficient neural processing but also to address complex problems, redefining the boundaries of neural computation.