Title:  A Feedback Methodology for Task-Driven Fine-Grained Pixel Control in Smart Cameras

Committee: 

Dr. Mukhopadhyay, Advisor   

Dr. Chatterjee, Chair

Dr. Krishna

Abstract:

The objective of the proposed research is to develop the concept of feedback control for smart camera systems driven by an end-task. Specifically, this camera system only captures useful information pertaining to an end-user defined task and at the highest quality. This is achieved through task-guided feedback provided by an embedded deep-neural network based task. The advantage of feedback is shown at the algorithm level, at the encoder level and at the level of the sensor itself. The feedback modifies the sampling characteristics of each individual pixel to achieve what is essentially task-guided information modulation. A number of control schemes and tasks are presented including spatial resolution control and temporal resolution control. Multiple use cases including object detection, object tracking, action detection, anomaly detection are demonstrated and their associated control schemes. Experiments show that the proposed schemes preserve or increase task accuracy while utilizing 3-4x lower bandwidth. Additionally, it is shown that for the feedback system, latency is a unique challenge particularly due to the high computational complexity of DNN networks. A simulation model for the feedback is developed to show how software/hardware solutions can be leveraged to deal with the latency challenge. Further, some qualitative challenges are presented encountered in unconstrained videos including small object detection and action detection in moving camera sequences. The remainder of the research will focus on developing an edge-host collaborative methodology and fast uncertainty estimation in the smart camera.