Title:  Analyzing Health-related Behaviors Using First-person Vision

Committee: 

Dr. Rehg,

Dr. Clements, Co-Advisor       

Dr. Inan, Chair

Dr. Abowd

Abstract:

The objective of the proposed research is to develop computation models for human behaviors analysis. Wearable devices are widely used in health-care industry and more personalized functions are expected by the practitioners and users. The signals from wearable devices not only record the physiological status of the subject, the environment they are exposed to, but also implicitly record their attention and intention. Extracting health-related info from the huge amount of data collect from various devices is a challenging topic in the research field and also promising one in health industry. This thesis focus on how to effectively analyze data collected from wearable devices.  We will start from videos recorded from wearable cameras to multi-modal signals. First, I present a model for first-person action recognition. This model leverage the shared motion and appearance properties of different actions. It is able to predict novel actions that the model is not trained on. Second, I develop a method to detect the screen-using moments of a wearable camera user. It can find out when the person is looking at a screen and localize the screen in the frame without a eye-tracker. Finally, I propose to work with multi-modal signals. I will build a deep neural network to predict the correspondence between video and acceleration. I want to find the time-offset to synchronize two signals and explore how the learned embedding space can be used to in cross-modal retrieval and activity recognition tasks.