Title:  Reliable Sensor Intelligence in Resource Constrained and Unreliable Environment

Committee:

Dr. Saibal Mukhopadhyay, ECE, Chair, Advisor

Dr. Tushar Krishna, ECE

Dr. Hyesoon Kim, CoC

Dr. Callie Hao, ECE

Dr. Justin Romberg, ECE

Abstract: The objective of this research is to design a sensor intelligence that is reliable in a resource constrained, unreliable environment. The research particularly focuses on improving the reliability of vision based intelligent sensor system using digital pixels to perform complex vision tasks such as image classification, object detection, and action detection. The research suggests three approaches to achieve this goal; (1) enhancing the task robustness, (2) adopting an early warning to predict task failure, and (3) employing uncertainty to estimate task reliability. First, to improve task robustness, the research studies a deep neural network (DNN) based fully spatiotemporal preprocessor that removes spatiotemporal corruptions of input data and enhances the spatiotemporal task performance. Moreover, a digital pixel-DNN cross-layer simulation methodology is presented to design a robust DNN to sensor hardware derived noise. Second, a concept of early warning generation is introduced to predict unreliable task in advance and avoid system failure. A DNN based early warning generator exploits spatiotemporal characteristics of input to predict the task failure due to low power sensor operation and controls sensor operation. Third, to adopt DNN model uncertainty as a measure of task reliability, lightweight uncertainty estimator is suggested that distills knowledge of a stochastic DNN and allows uncertainty quantification without prohibitive computation from stochastic DNN.