Title:  Towards Understanding the Purview of Neural Networks via Representation Learning

Committee: 

Dr. AlRegib, Advisor     

Dr. Davenport, Chair

Dr. Hoffman

Abstract: The objective of the proposed research is to devise novel representations that can be utilized to understand the purview of trained neural networks. Recent studies reveal that despite tremendous success of deep neural networks, they are prone to failure when deployed in real-world environments as they often encounter data that diverges from training conditions. This vulnerability stems from the limitation in activation-based representations. During inference, neural networks make predictions via forward propagations where they utilize features of different abstraction levels that they learned to extract from training data. While the activation-based decision making process may work in limited scenarios where inputs are of similar distribution as training data, they would suffer when presented with unfamiliar samples that their learned features are insufficient to properly represent. We propose to assess the scope of trained neural networks, prior to model deployment, by inspecting the amount of required model updates in response to diverse inputs. Specifically, we consider gradient-based representations generated in response to confounding labels—labels that are different from ordinary labels that a model in question is trained on. By introducing an unseen class label to a model with already defined representation space, the required model updates captured in gradients would be pertinent to mapping its relevant features to the new class. When the purview of the model is not broad enough for a given input, however, updates will be necessary for feature extraction as well as for feature mapping, leading to a larger total amount of updates. We present preliminary study on the gradient-based representations obtained with confounding labels, and we outline future work.