{"690525":{"#nid":"690525","#data":{"type":"news","title":"New Framework Enhances AR Experience by Predicting Where Users Will Look","body":[{"value":"\u003Cp\u003EAugmented reality (AR) devices like smart glasses may soon be able to predict where a user will look and provide an enhanced interactive experience.\u003C\/p\u003E\u003Cp\u003E\u003Ca href=\u0022https:\/\/fkryan.github.io\/\u0022\u003E\u003Cstrong\u003EFiona Ryan\u003C\/strong\u003E\u003C\/a\u003E, a Ph.D. student in Georgia Tech\u2019s School of Interactive Computing, is pioneering research that tracks and predicts user gaze from a first-person perspective in 3D environments.\u003C\/p\u003E\u003Cp\u003ECurrently, most AR devices react to where users look, playing catch-up. Ryan\u2019s method could give these devices a heads-up and make the user experience more seamless.\u003C\/p\u003E\u003Cp\u003E\u201cIt allows an AR system to anticipate what the person will interact with next and where they\u2019re going to look next so it can proactively render the experience,\u201d she said.\u003C\/p\u003E\u003Cp\u003ERyan is the lead author of the paper \u003Cem\u003EForecasting 3D Scanpaths in Egocentric Video,\u003C\/em\u003E which she will present next week at the\u0026nbsp;\u003Ca href=\u0022https:\/\/cvpr.thecvf.com\/\u0022\u003EIEEE Conference on Computer Vision and Pattern Recognition\u003C\/a\u003E (CVPR) in Denver.\u003C\/p\u003E\u003Cp\u003EWhile there is existing research on predicting user gaze from 2D still images, her work is the first to address the issue through a 3D framework.\u003C\/p\u003E\u003Cp\u003E\u201cBecause we live in a 3D world and people are dynamically moving around from multiple points of view, we need to predict gaze in 3D rather than 2D,\u201d she said. \u201cWhat we\u2019re seeing is a path of the person\u2019s attention in 3D through space. Our paper is the first to attempt to model this.\u201d\u003C\/p\u003E\u003Cp\u003ERyan conducted most of the research while interning at Meta, where she used data from Meta\u2019s\u0026nbsp;\u003Ca href=\u0022https:\/\/www.projectaria.com\/datasets\/adt\/\u0022\u003EAria Digital Twin dataset\u003C\/a\u003E. The dataset contains first-person video footage of users interacting with objects in an apartment.\u003C\/p\u003E\u003Cp\u003E\u201cWe chose that dataset because it has a high-fidelity 3D reconstruction of a full environment, which helps us get a ground-truth 3D gaze,\u201d she said. \u201cWe can trace eye movement and see how it intersects with the environment.\u201d\u003C\/p\u003E\u003Cp\u003EA video demonstration of Ryan\u2019s work shows her software tracking a user\u2019s path toward a table with a cup on it. Once the user picks up the cup, the software correctly predicts the direction the user will turn next.\u003C\/p\u003E\u003Cp\u003E\u201cWhen we look at a scene, we don\u2019t take in everything in full detail all at once,\u201d she said. \u201cWe fixate on certain areas, and our gaze is a sequence of fixations, which might depend on what we\u2019re trying to do. If we want to pick up a cup, we might look toward that and then the next step would be looking at where we\u2019re going to put it down.\u201d\u003C\/p\u003E\u003Cp\u003ERyan said the software can predict, on average, up to three seconds into the future \u2014 and as far as 10 seconds in some cases. That\u2019s enough time for the AR system to proactively render a more enhanced environment.\u003C\/p\u003E\u003Cp\u003E\u201cWe\u2019re not looking that far into the future right now, but it would be interesting to explore longer forecasting windows,\u201d she said. \u201cI think potential futures would diverge pretty quickly, so we\u2019re trying to explore what can reasonably be predicted from a short segment of a person looking and moving through space.\u201d\u003C\/p\u003E\u003Cp\u003ERyan said her paper served as a proof-of-concept, and that there is still much future work to be done. She already has some ideas.\u003C\/p\u003E\u003Cp\u003E\u201cI think future models can include different scenarios to help narrow down possibilities. Sometimes a person\u2019s gaze stays on one thing for a long time. If we know what someone is trying to do, we\u2019ll have a better idea of the likely path their attention might go.\u201d\u003C\/p\u003E\u003Cp\u003EThere could also be future implications for her work in robotics research.\u003C\/p\u003E\u003Cp\u003E\u201cIt could potentially be used for training algorithms for robots to emulate active human perception. If we can understand what a person looks at as they perform a task, we could use that to facilitate a robot learning to do that same task.\u201d\u0026nbsp;\u003C\/p\u003E","summary":"","format":"limited_html"}],"field_subtitle":"","field_summary":[{"value":"\u003Cp\u003EPh.D. student Fiona Ryan has created a new framework for tracking and predicting user gaze in Augmented Reality devices. If these devices know where a user will look next, it can proactively display information and interactive features more seamlessly.\u003C\/p\u003E","format":"limited_html"}],"field_summary_sentence":[{"value":"Ph.D. student Fiona Ryan has created a new framework for tracking and predicting user gaze in Augmented Reality devices"}],"uid":"36530","created_gmt":"2026-05-27 21:15:00","changed_gmt":"2026-05-27 21:16:17","author":"Nathan Deen","boilerplate_text":"","field_publication":"","field_article_url":"","location":"Atlanta, GA","dateline":{"date":"2026-05-27T00:00:00-04:00","iso_date":"2026-05-27T00:00:00-04:00","tz":"America\/New_York"},"extras":[],"hg_media":{"680364":{"id":"680364","type":"image","title":"IMG_2114.JPG","body":null,"created":"1779916518","gmt_created":"2026-05-27 21:15:18","changed":"1779916518","gmt_changed":"2026-05-27 21:15:18","alt":"Fiona Ryan","file":{"fid":"264620","name":"IMG_2114.JPG","image_path":"\/sites\/default\/files\/2026\/05\/27\/IMG_2114.JPG","image_full_path":"http:\/\/hg.gatech.edu\/\/sites\/default\/files\/2026\/05\/27\/IMG_2114.JPG","mime":"image\/jpeg","size":100549,"path_740":"http:\/\/hg.gatech.edu\/sites\/default\/files\/styles\/740xx_scale\/public\/2026\/05\/27\/IMG_2114.JPG?itok=uM3cBtPX"}}},"media_ids":["680364"],"groups":[{"id":"47223","name":"College of Computing"},{"id":"1188","name":"Research Horizons"},{"id":"50876","name":"School of Interactive Computing"}],"categories":[{"id":"153","name":"Computer Science\/Information Technology and Security"},{"id":"135","name":"Research"}],"keywords":[{"id":"188776","name":"go-research"},{"id":"187915","name":"go-researchnews"},{"id":"9153","name":"Research Horizons"},{"id":"1597","name":"Augmented Reality"},{"id":"11506","name":"computer vision"},{"id":"183308","name":"smart glasses"}],"core_research_areas":[{"id":"39501","name":"People and Technology"}],"news_room_topics":[],"event_categories":[],"invited_audience":[],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[],"email":[],"slides":[],"orientation":[],"userdata":""}}}