Title:  Distributed Learning and Inference in Deep Models

Committee:

Dr. Faramarz Fekri, ECE, Chair , Advisor

Dr. Ghassan AlRegib, ECE

Dr. Justin Romberg, ECE

Dr. Matthieu Bloch, ECE

Dr. Siva Theja Maguluri, ISyE

Abstract: In this thesis, we consider the challenges encountered in training and inference of large deep models, especially on nodes with limited computational power and capacity. We study two classes of related problems; 1) distributed training of deep models, and 2) compression and restructuring of deep models for efficient distributed and parallel execution to reduce inference times. Especially, we consider the communication bottleneck in distributed training and inference of deep models. In the first part of the thesis, we consider distributed deep learning. Data compression is a viable tool to mitigate the communication bottleneck. However, the existing methods suffer from a few drawbacks, such as the increased variance of stochastic gradients (SG), slower convergence rate, or added bias to SG. We address these challenges from three different perspectives: 1) Information Theory and the CEO Problem, 2) Indirect SG compression via Matrix Factorization, and 3) Compressive Sampling.

Next, we consider federated learning over wireless multiple access channels (MAC). To satisfy the communication and power constraints of the network, and take advantage of the over-the-air computation inherent in MAC, we propose a framework based on random linear coding and develop efficient power management and channel usage techniques to manage the trade-offs between power consumption and communication bit-rate. In the second part of this thesis, we consider the distributed parallel implementation of an already-trained deep model on multiple workers. Since latency due to the synchronization and data transfer among workers adversely affects the performance of the parallel implementation, it is desirable to have minimum interdependency among parallel sub-models. To achieve this goal, we introduce RePurpose, an efficient algorithm to rearrange the neurons in the neural network and partition them such that the interdependency among sub-models is minimized under the computations and communications constraints of the workers.