674377 event 1714042640 1714042791 <![CDATA[Ph.D. Proposal Oral Exam - Nealson Li]]> Title:  Eye Tracking and Gaze Estimation with Event Camera for Extended Reality

Committee: 

Dr. Raychowdhury, Advisor    

Dr. Krishna, Chair

Dr. Sathe

]]> The objective of the proposed research is to develop an algorithm-hardware co-designed system capable of precise eye tracking and gaze estimation using an event camera. Eye tracking is an essential functionality to enable extended reality (XR) applications. However, the latency and power constraints of an XR headset are tight. Unlike fix-rate frame-based RGB cameras, the event camera senses brightness changes and generates asynchronous sparse events with high temporal resolution. Although the event camera exhibits suitable characteristics for eye tracking in XR systems, processing an event-based data stream is a challenging task. We present an event-based eye-tracking system that extracts pupil features and is able to locate the pupil with an error of 3.68 pixels at 160 mW system power. We also proposed a system that achieves accurate real-time gaze estimation with an angular accuracy of 0.46° and update rates of 950 Hz. To our knowledge, these are the first systems that operate only on event-based data to perform eye tracking and gaze estimation. Future work aims to explore graph representation methodologies for event-based data streams and design an GNN inference accelerator to optimize efficiency and tackle the challenges of irregular and dynamic operations.  The completed system should leverage the inherent advantages of event-based data stream to provide exceptional performance and energy efficiency and ultimately contributing to advancements in XR.

]]> <![CDATA[]]> https://teams.microsoft.com/l/meetup-join/19%3ameeting_ZTRhNDE5YjEtMDlhOS00YjUxLTkwYzQtNjQzMTBiZTQ1NDM2%40thread.v2/0?context=%7b%22Tid%22%3a%22482198bb-ae7b-4b25-8b7a-6d7f32faa083%22%2c%22Oid%22%3a%22bd7e1212-2d0e-43f1-a241-e796ac426fff%22%7d 434371 1788 102851 1808