event

PhD Defense by Hyeokhyen Kwon

Primary tabs

Title: Deriving Sensor-based Complex Human Activity Recognition Models Using Videos

 

Hyeokhyen Kwon

Ph.D. Candidate in Computer Science

School of Interactive Computing

College of Computing

Georgia Institute of Technology

 

Date: Monday, September 20th, 2021

Time: 11AM – 2PM (EST)

Location: https://bluejeans.com/3537894193

 

Committee:

Dr. Gregory D. Abowd (Advisor) – Electrical and Computer Engineering, Northeastern University

Dr. Thomas Plötz (Co-Advisor) – School of Interactive Computing, Georgia Institute of Technology

Dr. Thad Starner – School of Interactive Computing, Georgia Institute of Technology

Dr. Irfan Essa – School of Interactive Computing, Georgia Institute of Technology

Dr. Nicholas D. Lane – Dept. of Computer Science & Tech., University of Cambridge

 

Abstract:

Inertial Measurement Unit (IMU)-based human activity recognition (HAR) is at the core of continuous monitoring for human well-being, detecting precursors of health risks in everyday life. HAR systems are commonly developed with supervised learning techniques with labeled sensor datasets. Thanks to the advances in supervised learning techniques, such as deep learning models, HAR recently have experienced several successes in improving recognition performances. However, the progress of HAR still has been facing challenges mainly due to the difficulty of collecting sensor data, since large-scale datasets are required to effectively optimize sophisticated deep learning models. Conventionally, wearable sensor data is collected from recruited users, where user engagement is expensive and the annotation is time-consuming. Thereby, the collected sensor dataset typically remains in a small-scale, which has been constraining HAR models to increase the complexity of deep learning models to effectively analyze the sensor data.

To overcome the limitations of HAR models, this dissertation presents major steps towards the greater vision of automated large-scale data collection for body-worn sensing systems. I  propose a novel approach to tackle the scale limitations in the wearable sensor dataset, which aims to use existing video data from virtually unlimitedly large repositories, such as YouTube, as a novel source of the training dataset. IMU sensors essentially capture the human motion information of ongoing activities. The proposed system bridges the modality gap between videos and on-body IMU sensors through an automated processing pipeline that integrates existing computer vision, graphics, and signal processing techniques to convert human activity videos into virtual IMU data streams. The proposed system automatically handles numerous challenges in unconstraint online videos for tracking 3D human motion to generate a high-quality virtual IMU dataset. I show how the virtually-generated IMU data improves the performance of various models on known human activity recognition datasets. With the benefit of collecting large-scale training datasets from online videos, I demonstrate that very complex models can effectively analyze sensor datasets, which was infeasible previously with a small-scale dataset. Through a series of experiments, this dissertation shows the promise of using video as a novel source for human activity recognition with wearables, representing a paradigm shift for deriving a robust human activity recognition system.

 

Status

  • Workflow Status:Published
  • Created By:Tatianna Richardson
  • Created:09/07/2021
  • Modified By:Tatianna Richardson
  • Modified:09/07/2021

Categories

Keywords