677502 event 1728688096 1728688209 <![CDATA[Ph.D. Proposal Oral Exam - Uday Kamal]]> Title:  Efficient processing of temporally asynchronous and spatially unstructured event-based camera data

Committee: 

Dr. Mukhopadhyay, Advisor    

Dr. Hao, Chair

Dr. Romberg

]]> The objective of the proposed research is to address the challenge of efficiently processing ultra-high-speed (order of μs temporal resolution) event-based camera data. Processing event-based camera data presents significant challenges due to the asynchronous and sparse nature of the events. Traditional image processing methods, designed for frame-based data, struggle with the high temporal resolution and spatially unordered event streams that event cameras produce. Efficiently handling these data streams, especially for high-speed perception tasks, requires innovative approaches that can process events in real-time without compromising on accuracy or computational efficiency. To address these challenges, we first propose a novel event-based representation learning framework, Eventformer, that processes asynchronous event data by treating input events as a spatially unordered streaming set of inputs. It leverages a self-attention mechanism to model spatial interactions among events and uses an associative memory-augmented recurrent module to correlate these events with stored representations. A novel memory addressing mechanism allows for the storage and retrieval of latent states only at event locations, updating them dynamically as new events occur. This shift in representation learning from the input space to a latent memory space significantly reduces computational costs without sacrificing the algorithmic performance on standard event-based object recognition benchmarks [1]. Building on this, we further develop a spatiotemporal representation learning framework tailored for event-based dense prediction tasks, which require pixel-level associations—a particularly difficult requirement for event-based data. Our approach uniquely handles sparse, asynchronous events through a set-based method, projecting them into a hierarchically organized, multi-level latent memory space that preserves pixel-level structure. Low-level event streams are dynamically encoded into these latent structures using an attention-based spatial association. Unlike traditional methods that update memory stacks at a fixed rate, our framework introduces a data-adaptive update mechanism that selectively updates memory stacks only when significant new information is detected. This approach dramatically improves compute latency while maintaining competitive performance across various event-based dense prediction tasks [2]. In our ongoing (and future) works, we aim to apply and adapt our proposed framework for more challenging tasks, including high speed autonomous navigation while avoiding obstacles and spatiotemporal location of high-speed action. In summary, this thesis aims to develop a novel memory augmented representation learning pipeline for efficient processing of event-based camera data and leverage it for diverse event-based perception tasks. The proposed research would advance the processing of event-based camera data, offering efficient, real-time capabilities for both object recognition and dense prediction tasks, with significant reductions in computational overhead and latency.

 

]]> <![CDATA[]]> https://gatech.zoom.us/j/99452254067?pwd=N4KDYEOraYlxjniF1yLNGyAUtVskoa.1 434371 1788 102851 1808