The surrounding trees resemble those of a eucalyptus forest, which could indicate northern Portugal or the Spanish region of Galicia.

It’s the signposts on the road that give it away. They are flat and wide, which is common in Spain but not in Portugal.

Casapu, a master’s student in Georgia Tech’s School of City and Regional Planning, correctly reasons that the picture of a road he’s looking at is in Galicia.

Give Casapu a photo, and he will likely be able to tell you where it was taken. 

“I start with infrastructure clues that are specific to a country, region, state or province,” Casapu said. “They include roads or electricity poles, which often remain consistent throughout a country. Once you narrow down the country, you can use more specific factors like vegetation, specific landscapes, or architecture, because these are very nuanced. It’s a top-down approach.”

This is why Casapu is the reigning GeoGussr World Champion — and the ideal expert to test vision-language models (VLMs) on how good they are at geolocation.

GeoGuessr is a geography browser game launched in 2013 that invites players to guess the location of random Google Street View images. Casapu was already known as one of the top players in the world before he won the third annual GeoGussr World Championship in September.

At the beginning of the spring 2025 semester, School of Interactive Computing professor James Hays reached out to Casapu and invited him to collaborate on a new project. Hays was looking to create a dataset to evaluate VLMs' geolocation ability and reasoning. 

“VLMs are surprisingly good at geolocation right out of the box, even when they’re not trained to be good at it,” Hays said.

Hays and his colleagues, associate professors Alan Ritter and Wei Xu, took issue with many AI companies claiming that the VLMs they were releasing were not good at geolocation.

“When Open AI released GPT 4 Vision, there were privacy concerns about the model’s ability to geolocate someone based on photos they’ve shared on the internet,” Ritter said. “Open AI said this wasn’t a concern and claimed the model wasn’t good at geolocation beyond being able to recognize a city or famous monument. We found that wasn’t the case. These VLMs are state-of-the-art at image geolocation tasks.”

Show Your Work

Hays and Ritter enlisted a team of some of the world’s top geolocators. It consisted of Casapu, Joshua Diao, a master’s student in computer science, and Tejas Santanam, a Ph.D. student in industrial engineering. They each received 500 images to geolocate.

Team members recorded their reasons for each of their answers. The result was GeoRC, the first benchmark for VLM geolocation performance, consisting of 800 “ground truth” reasoning chains. 

Hays and Ritter gave the same images to GPT 5, Gemini, Llama, and Qwen. The highest-performing model geolocated with 90% accuracy — not far off from the team’s 96% score.

However, a major distinction showed up in the reasoning chains. While Casapu and Diao provided clear explanations for how they deduce each location, the VLMs either couldn’t provide reasoning for their guesses or were vague in their answers.

“The research community has been demanding explanations from these models,” Hays said. “For example, how do they know the location is in Italy?”

Hays has been researching this subject for almost 20 years. As a Ph.D. student at Carnegie Mellon University in 2008, he was the first researcher to take a machine learning approach to geolocation. He introduced a new algorithm that could estimate a geographic location from a single image.

“When experts have audited these reasoning chains, we’ve noted many suspicious or hallucinated attributes,” he said. “When they hallucinate a geographic property, why is it so often consistent with the correct guess?

“I believe they’re not revealing the true reasoning pathway that they used to determine the image was Italy. They’re just implicitly recognizing that it was Italy for many reasons, then hunting for evidence to support that. Some of the things they say are true and supported by the image, and some are fabrications.”

Practice Partner

Casapu said there may be only a handful of GeoGuessr players who can currently outperform some top-tier VLMs in geolocation, and it may not be long before no one can.

“I think it could be more difficult playing against these models than playing against another human because a human has the possibility of making mistakes at the top level,” Casapu said. “If a well-trained model has that level of consistency, that is far beyond a normal person, and it would be much more difficult to beat.”

He added that working with Hays and competing against a machine improved his skill level and provided valuable practice ahead of the world championship. 

“It helps to take a step back and see why you’re making the guesses that you are,” he said. “Since then, I’ve taken a more methodical approach to how I practice. Writing these things down is a great way to see what you know and see why you make the guesses that you do. It’s been a great training tool.”

Casapu will defend his title at the 2026 GeoGussr World Championship in September.

Hays, Ritter, Xu, Casapu, Diao, and Santanm are all co-authors of a paper on GeoRC along with lead author Mohit Talrej and Ph.D. students Ethan Mendes and Jim Thannikary. The paper will be presented next week at the 64th Annual Meeting of the Association for Computational Linguistics (ACL) in San Diego.

RAI calls the bicycle robot an ultra-mobility vehicle (UMV). It can reach a height of 3 feet and can jump from the floor onto a platform.

The contributions of a Georgia Tech Ph.D. student helped make these feats possible through a robot control policy he developed.

Jeonghwan Kim, who is pursuing a Ph.D. in robotics under the advisement of Associate Professor Sehoon Ha, spent two semesters interning at RAI. His task was to design a policy to teach the UMV to land after a flip.

The result was iterative motion imitation (IMI), a novel method that imitates flip trajectories generated from prior examples. Kim said the robot bases its flip on a demonstration, and human engineers reconstruct and refine the flip path through simulation to fill in the gaps.

“To guide the robot to flip, we started with an imperfect trajectory generated by a motor-based controller and then ran simulations,” Kim said. “It’s an unstable trajectory, but we use it as a guide to train a single policy that can track it as it lands and tries to balance itself.”

Sticking the Landing

Kim interned under the supervision of Shamel Fahmi, a research scientist at the RAI Institute. RAI has been developing the UMV for nearly three years.

“We wanted to work on a different robot morphology that wasn’t legs or legs with wheels,” Fahmi said. “That’s when we thought of working with bikes. 

“We want to merge the athleticism of (Boston Dynamics’) Atlas with the mobility of a bike. We wanted a robot that can go anywhere, do parkour, and acrobatics.”

Fahmi said that before Kim arrived, the research team had trouble getting the UMV to land consistently without breaking or falling.

The UMV has two joints — an upper and a lower. The upper joint contains the motors and pulls the lower joint along as it propels into the air. The problem is getting the lighter lower joint to absorb the impact of landing without being crushed by the heavier upper joint.

“That’s what brings reinforcement learning into the equation,” Fahmi said. “We teach the robot to minimize its impact on the ground to land gracefully.”

Fahmi said that Kim proved the imitation examples the robot learns from don’t have to be perfect. The process takes some time, but all it needs is a rough idea to get started.

“You can have an imperfect sketch and then constantly refine it,” Fahmi said. “The first time, it’s not going to go well. 

“We don’t care about torque or power limits as long as it does the motion. Then we’ll have a slightly better reference, repeat it, and imitate it again. In every iteration, we can add more parameters.”

Up Against the Clock

Kim said he felt the pressure of time constraints during his two semesters with RAI as he worked to achieve consistent, successful landings. Even though he had multiple UMVs to experiment with, they broke down dozens of times. Each time one broke, a hardware team at RAI had to repair it.

“There was a lot of pressure to not only get this working before my internship ended, but also knowing there are costs behind every failed attempt, and every time the robot breaks, it takes time to repair it,” Kim said. 

“It took almost five months for it to land without breaking. Then we needed two more months for it to stay balanced after the landing. It requires a lot of engineering effort to achieve a robust control policy for a safe flip.”

By the time Kim left RAI, the IMI policy had achieved consistent, seamless landings.

“The jump right now is what we call the visitor demo,” Fahmi said. “If there are guests coming over to see it, we want to show them something that is extremely impressive, but also, more importantly, extremely reliable. It never fails.

“It was only possible because of the huge effort we put into designing, maintaining, and continuously improving the robot.”

Kim authored a paper on his framework and will present it at this week’s International Conference on Robotics and Automation (ICRA) in Vienna.

For more information about the UMV project, please visit the RAI blog or watch their video on YouTube.

The new technology is also set to revolutionize the scale of policy training data collection, which is essential to advancing robotic capabilities and meeting growing production demand.

COBALT is a mobile app that turns smartphones into controllers for robot arms. With a secure Wi-Fi connection to a server, users can move their phones in any direction, and the robot arm will mirror the motion — from anywhere in the world.

Ayush Agarwal, a Ph.D. student in Georgia Tech’s School of Interactive Computing who leads a research team developing COBALT, said it works like the games people play on smartphones. Users can press a button to have the arm grasp an object, move it, and release it with another button.

Agarwal conducted several user studies with participants in nine countries who remotely operated robot arms inside Georgia Tech’s People, AI & Robotics (PAIR) Lab. The lab is directed by Assistant Professor Animesh Garg, who advises Agarwal.

“We built an entire distribution system for remote teleoperation scaled to where we had people from Indonesia, India, and Pakistan operating for us,” Agarwal said. “They were novice operators who had never done it before. By collecting data from these new users, we showed that we can train policies to automate certain tasks.”

Garg envisions a world where data collection for policy training is done through crowdsourcing. He began working toward this goal 10 years ago as a postdoc at Stanford University, when he developed RoboTurk, an earlier version of COBALT.

“There is a large-scale data collection requirement for mass robot production to be possible, and it will not be solved purely through simulation,” Garg said.

“Our idea was, what if we could get almost every person on the planet to be a passive source for data collection? There are almost five billion people who have smartphones and know how to use them.”

Education and Economy Impact

Another major implication of COBALT could be expanded access to CS and robotics education.

Students can learn to operate a robot remotely in any classroom. In fact, Garg and his lab recently hosted students from Midtown High School in Atlanta to demonstrate COBALT and let them control robot arms from a phone.

Garg also sees the possibility of a “gig economy” in which people pay remote operators to control assistive robots in their homes and complete household chores for them.

“It could be Uber for robots,” he said. “People who want to log onto the platform can do so at their convenience and for as long as they want.”

Companies with robot-dependent labor tasks could also use the platform to enable human oversight.

“If I deploy a robot in a factory that achieves high autonomy for most tasks, but there are still times it needs help, a human could operate the robot from anywhere in the world,” Garg said.

Building a Network

Agarwal’s studies showed that people prefer to interact with and control a robot using a smartphone rather than virtual reality (VR) headsets, controllers, keyboards, mice, or other devices.

“The phone is a more intuitive interface and can provide data quality that’s on par with other commonly used devices,” he said.

Agarwal also said there is minimal latency in the video feed sent back to operators on the other side of the world. That’s because the amount of data being processed is small.

The data is carried over Web Real-Time Communication (WebRTC), the same technology used by many streaming services and web conferencing platforms such as Zoom and Google Meet.

“There’s a connection from your phone to the teleoperation server, which is connected to the robots,” Agarwal said.

“Then there’s another connection from the teleoperation server back to the user, which allows for a video stream. We need low latency on both because you don’t want the user to move their phone and wait 10 seconds to see the visual feed.”

Agarwal is the co-lead author of a paper on COBALT that is being presented at the IEEE International Conference on Robotics and Automation this week in Vienna. He said the paper stands out because it has moved from theory to the implementation of an entire distribution network. 

“The real novelty of our paper is the systems that we build around it to actually support the scaling of remote operation and data collection at a global level,” he said. 

Fiona Ryan, a Ph.D. student in Georgia Tech’s School of Interactive Computing, is pioneering research that tracks and predicts user gaze from a first-person perspective in 3D environments.

Currently, most AR devices react to where users look, playing catch-up. Ryan’s method could give these devices a heads-up and make the user experience more seamless.

“It allows an AR system to anticipate what the person will interact with next and where they’re going to look next so it can proactively render the experience,” she said.

Ryan is the lead author of the paper Forecasting 3D Scanpaths in Egocentric Video, which she will present next week at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in Denver.

While there is existing research on predicting user gaze from 2D still images, her work is the first to address the issue through a 3D framework.

“Because we live in a 3D world and people are dynamically moving around from multiple points of view, we need to predict gaze in 3D rather than 2D,” she said. “What we’re seeing is a path of the person’s attention in 3D through space. Our paper is the first to attempt to model this.”

Ryan conducted most of the research while interning at Meta, where she used data from Meta’s Aria Digital Twin dataset. The dataset contains first-person video footage of users interacting with objects in an apartment.

“We chose that dataset because it has a high-fidelity 3D reconstruction of a full environment, which helps us get a ground-truth 3D gaze,” she said. “We can trace eye movement and see how it intersects with the environment.”

A video demonstration of Ryan’s work shows her software tracking a user’s path toward a table with a cup on it. Once the user picks up the cup, the software correctly predicts the direction the user will turn next.

“When we look at a scene, we don’t take in everything in full detail all at once,” she said. “We fixate on certain areas, and our gaze is a sequence of fixations, which might depend on what we’re trying to do. If we want to pick up a cup, we might look toward that and then the next step would be looking at where we’re going to put it down.”

Ryan said the software can predict, on average, up to three seconds into the future — and as far as 10 seconds in some cases. That’s enough time for the AR system to proactively render a more enhanced environment.

“We’re not looking that far into the future right now, but it would be interesting to explore longer forecasting windows,” she said. “I think potential futures would diverge pretty quickly, so we’re trying to explore what can reasonably be predicted from a short segment of a person looking and moving through space.”

Ryan said her paper served as a proof-of-concept, and that there is still much future work to be done. She already has some ideas.

“I think future models can include different scenarios to help narrow down possibilities. Sometimes a person’s gaze stays on one thing for a long time. If we know what someone is trying to do, we’ll have a better idea of the likely path their attention might go.”

There could also be future implications for her work in robotics research.

“It could potentially be used for training algorithms for robots to emulate active human perception. If we can understand what a person looks at as they perform a task, we could use that to facilitate a robot learning to do that same task.” 

Fiona Ryan, a Ph.D. student in Georgia Tech’s School of Interactive Computing, is pioneering research that tracks and predicts user gaze from a first-person perspective in 3D environments.

Currently, most AR devices react to where users look, playing catch-up. Ryan’s method could give these devices a heads-up and make the user experience more seamless.

“It allows an AR system to anticipate what the person will interact with next and where they’re going to look next so it can proactively render the experience,” she said.

Ryan is the lead author of the paper Forecasting 3D Scanpaths in Egocentric Video, which she will present next week at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in Denver.

While there is existing research on predicting user gaze from 2D still images, her work is the first to address the issue through a 3D framework.

“Because we live in a 3D world and people are dynamically moving around from multiple points of view, we need to predict gaze in 3D rather than 2D,” she said. “What we’re seeing is a path of the person’s attention in 3D through space. Our paper is the first to attempt to model this.”

Ryan conducted most of the research while interning at Meta, where she used data from Meta’s Aria Digital Twin dataset. The dataset contains first-person video footage of users interacting with objects in an apartment.

“We chose that dataset because it has a high-fidelity 3D reconstruction of a full environment, which helps us get a ground-truth 3D gaze,” she said. “We can trace eye movement and see how it intersects with the environment.”

A video demonstration of Ryan’s work shows her software tracking a user’s path toward a table with a cup on it. Once the user picks up the cup, the software correctly predicts the direction the user will turn next.

“When we look at a scene, we don’t take in everything in full detail all at once,” she said. “We fixate on certain areas, and our gaze is a sequence of fixations, which might depend on what we’re trying to do. If we want to pick up a cup, we might look toward that and then the next step would be looking at where we’re going to put it down.”

Ryan said the software can predict, on average, up to three seconds into the future — and as far as 10 seconds in some cases. That’s enough time for the AR system to proactively render a more enhanced environment.

“We’re not looking that far into the future right now, but it would be interesting to explore longer forecasting windows,” she said. “I think potential futures would diverge pretty quickly, so we’re trying to explore what can reasonably be predicted from a short segment of a person looking and moving through space.”

Ryan said her paper served as a proof-of-concept, and that there is still much future work to be done. She already has some ideas.

“I think future models can include different scenarios to help narrow down possibilities. Sometimes a person’s gaze stays on one thing for a long time. If we know what someone is trying to do, we’ll have a better idea of the likely path their attention might go.”

There could also be future implications for her work in robotics research.

“It could potentially be used for training algorithms for robots to emulate active human perception. If we can understand what a person looks at as they perform a task, we could use that to facilitate a robot learning to do that same task.” 

Mizan Rahman knows there’s much that academia and industry can learn from each other.

He’s living proof of it.

The 52-year-old entrepreneur will receive his Ph.D. in human-centered computing (HCC) as he walks across the stage on Thursday at Georgia Tech’s Spring 2026 Ph.D. Commencement.

When Rahman was accepted into the HCC Ph.D. program, he’d already founded three successful tech startups and was an angel investor in numerous others. He also earned a master’s in computational science and engineering from Georgia Tech in 2013.

Rahman took on the challenge of a Ph.D. because he’s always been in pursuit of a holistic view of technology. One perspective he said he needed to understand was that of the end user.

“I’d already done computer science and computational science and engineering, so I wanted to look at the human dimension, the user’s perspectives, and society,” Rahman said. “You’ve got to build technology that fits into our human dynamics.”

Rahman’s journey began as an undergraduate in chemical engineering at Miami Dade College and Florida Atlantic University. He switched to computer science after his roommate, also a CS major, showed him some programming he had been working on.

“I couldn’t sleep after that,” Rahman said. “I was writing software all night. I loved solving problems through technology.”

Early Success

Rahman invented BayBuilder, a strategic sourcing automation technology, in 1999. The software was adopted by major Fortune 500 companies. Rahman estimates it has saved these companies $1 billion in procurement spending.

Baybuilder was acquired by a NASDAQ-listed firm in 2001, and he was ready to start his next company.

“I’ve been an entrepreneur as far back as I can remember,” Rahman said. “I was born with it. If I saw something that didn’t exist, I created it.”

After relocating to Atlanta, Rahman founded a new company, M2SYS Technology. Governments around the world used the company’s innovative identity technology to automate processes and deliver efficient services to citizens. M2SYS also worked with the CDC to treat HIV in Haiti and Zambia, as well as many U.S. hospitals, including Grady Memorial in Atlanta, to protect patients from fraud and receiving the wrong treatment.

Rahman’s most recent startup, CloudApper AI, introduced a new system architecture that generates secure software requiring minimal ongoing maintenance. His non-biased algorithm, which he created during his Ph.D. for CloudApper, is now used by major companies to streamline automated resume analysis and candidate scoring.

Living in Two Worlds

Rahman began his Ph.D. in 2021, but he kept his new venture to himself and his family. He didn’t tell his employees he was pursuing a Ph.D., and he didn’t disclose his industry background to his fellow doctoral students.

“I kept the other side of me far away,” he said. “The people who knew, they knew, but I purposefully didn’t discuss my outside activities and experience. I wanted to fit in, and I think I was able to do that.”

When Rahman was at his company, he was a CEO and entrepreneur, and when he was at Georgia Tech, he was a researcher. But what he was learning as a researcher began to change how he perceived his business. 

“I wanted to be a researcher and think like a researcher and not just always think about sales and marketing,” he said. “I started bringing in more ideas about how the user should be thought of in our products. I’m sure they were wondering why I was emphasizing that so much, but it was because I was applying what I was learning in my Ph.D. 

“Now I’ve been on both sides, I want to be connected to both in the future, applying research principles and practices in product development and innovation.”

Building Community Through Makerspaces

When it came time for Rahman to choose a subject for his dissertation, he returned to his roots and looked for ways technology can support young entrepreneurs and their startups. That’s when he began conducting research in makerspaces.

“I wanted to find out how we can bring innovation to a scale where anybody can participate,” he said. “I saw this happening in makerspaces where regular people learn, collaborate, and build products and companies from scratch. I saw that the community at large is facing a sustainability crisis.”

Rahman argued in his dissertation that makerspaces can play a significant role in local innovation. When people struggle to survive, it disrupts communities in numerous ways.

Rahman details four studies conducted over three-and-a-half years that show how socio-technical factors drive organizational sustainability in makerspaces and how AI tools can foster an innovative culture within them.

“The compelling thing about his research is that he shows that people come to makerspaces for the tools, but they stay for the people,” said Rosa Arriaga, associate professor and Rahman’s advisor.

“He has plenty of work from his ethnographic research that shows that a makerspace can have all the tech and resources, but if there isn’t cohesion among the people, there’s a problem.”

It Takes a Village

Rahman is the first to admit that it’s not possible for one man to run a company while pursuing a Ph.D. He needed a community. This starts with his family. His wife, Mohu Sultana, now serves as interim CEO of M2SYS and has supported Rahman throughout his Ph.D. research.

The Georgia Tech community has been part of Rahman’s life in some way since he started his career. 

Sultana holds a bachelor’s degree in computer science from Tech, and their daughter, Malisha Rahman, is graduating this week with a bachelor’s in economics and international affairs. Malisha Rahman has also been accepted into the HCC program and will begin her Ph.D. in the fall. 

Rahman said that any student who wants to create a tech startup will have an advantage from access to Georgia Tech’s network.

“The Georgia Tech startup community is fantastic,” he said. “There is a tremendous amount of knowledge here, and the research community can help shape the next big thing. We have CREATE-X, a place where you can find mentorship from faculty who started in industry. You’ll learn things I wish I knew before I started.”

That’s one of the main findings from a study by AI Caring on what older adults expect from explainable AI (XAI).

AI Caring is one of three AI Institutions led by Georgia Tech and funded by the National Science Foundation (NSF). The institution supports AI research that benefits older adults and their caregivers.

Niharika Mathur, a Ph.D. candidate in the School of Interactive Computing, was the lead author of a paper based on the study. The paper will be presented in April at the 2026 ACM Conference on Human Factors in Computing Systems (CHI) in Barcelona.

Mathur worked with the Cognitive Empowerment Program at Emory University to interview 23 older adults who live alone and use voice-activated AI assistants like Amazon’s Alexa and Google Home.

Many of them told her they feel excluded from the design of these products.

“The assumption is that all people want interactions the same way and across all kinds of situations, but that isn’t true,” Mathur said. “How older people use AI and what they want from it are different from what younger people prefer.”

One example she gave is that young people tend to be informal when talking with AI. Older people, on the other hand, talk to the agent like they would a person.

“If Older adults are talking to their family members about Alexa, they usually refer to Alexa as ‘she’ instead of ‘it,’” Mathur said. “They tend to humanize these systems a lot more than young people.”

Good Explanations

The study evaluated AI explanations that drew information from four sources of data:

• User history (past conversations with the agent)
• Environmental data (indoor temperature or the weather forecast)
• Activity data (how much time a user spends in different areas of the home)
• Internal reasoning (mathematical probabilities and likely outcomes)

Mathur said older users trust the agent more when it bases its explanations on data from the first three sources. However, internal reasoning creates skepticism.

Internal reasoning means the AI doesn’t have enough data from the other sources to give an explanation. It provides a percentage to reflect its confidence based on what it knows.

“The overwhelming response was negative toward confidence scores,” Mathur said. “If the AI says it’s 92% confident, older adults want to know what that’s based on.”

This is another example that Mathur said points to generational preferences.

“There’s a lot of explainable AI research that shows younger people like to see numbers in explanations, and they also tend to rely too much on explanations that contain numerical confidence. Older adults are the opposite. It makes them trust it less.”

Knowing the Context

Mathur said that AI agents interacting with older adults should serve a dual purpose. They should provide users with companionship and support independence while reducing the caretaking burden often placed on family members. 

Some studies have shown that engineers have tended to favor caretakers in the design of these tools. They prioritize daily tasks and routines, leaving some older adults to feel like they are merely a box to be checked.

She discovered that in urgent situations, older users prefer the AI to be straightforward, while in casual settings, they desire more conversation.

“How people interact with technological systems is grounded in what the stakes of the situation are,” she said. “If it had anything to do with their immediate sense of safety, they did not want conversational elaboration. They want the AI to be very direct and factual.”

Not Just Checking Boxes

Mathur said AI agents that interact with older adults are ideally constructed with a dual purpose. They should provide companionship and autonomy for the users while alleviating the burden of caretaking that is often placed on their family members. 

Some studies have shown that engineers have strayed toward favoring caretakers in the design of these tools. They prioritize daily tasks and routines, leaving some older adults to feel like they are a box to be checked.

“They’re not being thought of as consumers,” Mathur said. “A lot of products are being made for them but not with them.”

She also said psychological well-being is one of the most important outcomes these tools should produce. 

Showing older adults that they are listened to can significantly help in gaining their trust. Some interviewees told Mathur they want agents who are deliberate about understanding their preferences and don’t dismiss their questions.

Meeting these needs reduces the likelihood of protesting and creating conflict with family members.

“It highlights just how important well-designed explanations are,” she said. “We must go beyond a transparency checklist.”

They would normally get this in their natural habitats while foraging for food and staying alert to predators that might target calves.

However, the four elephants reside at Zoo Atlanta, so they don’t have to worry about these things.

That’s why zoo caretakers are always on the lookout for better ways to help their elephants exercise their brains.

The caretakers at Zoo Atlanta found one when they met Arianna Mastali, a Ph.D. student in Georgia Tech’s School of Interactive Computing. Mastali designed an audio enrichment wall to help stimulate Zoo Atlanta’s elephants.

Many zoos build concrete enrichment walls to foster elephant problem-solving and critical thinking. The walls usually have holes for the elephants to reach through with their trunks as they search for food, treats, or playful objects on the other side.

Mastali enhanced Zoo Atlanta’s enrichment wall by adding an interactive audio component. A nearby speaker system emits distinctive low-frequency tones when an elephant sticks its trunk into a hole.

“They’re intelligent creatures that require a lot of complexity in their habitat,” Mastali said. “We wanted to add to that complexity while giving them more control.”

Experimenting in the Wild

Mastali’s system uses cameras and computer vision to detect when an elephant’s trunk is inside a hole and then sends a signal to the speakers to play a sound.

Mastali is a member of the Georgia Tech Animal Lab, directed by School of IC professor Melody Jackson. The lab often uses sensing technology to enhance animal wellness.

Mastali said she tried incorporating sensing devices into her project several times. She constructed an insert made of PVC pipe and attached a sensor to its base that used infrared beams to detect the elephant’s trunk.

However, she said it was difficult to account for the elephants’ strength. Their trunks would break the insert after a day or two. 

She pivoted toward computer vision to remove the risk of damage and keep the enrichment wall as close to natural as possible. 

“A big lesson we learned was that using existing materials the elephants are already familiar with was the best way to do things, and it simplified our design process,” she said.

Shane Rosse, a student in Georgia Tech’s Online Master of Science in Computer Science (OMSCS) program, assisted Mastali with the computer vision component.

Enhancing Environmental Enrichment

Mastali observed the elephants’ behavior at the wall seven days before and seven days after the installation of the audio enrichment system.

The number of times the elephants approached the wall after installation increased by 176%, and time spent at the wall increased by 71%

“We weren’t sure at first if they would care that much, so it was great to see how much time they spent at the wall, especially our less dominant females,” said Kirby Miller, senior elephant caretaker at Zoo Atlanta. “They seem to like it the most.”

Miller said the elephants used to only approach the wall when they knew there was food behind it. That started to change after the audio enrichment system was installed.

“We would be off somewhere else, and we’d hear the speaker playing the sounds, and we knew there wasn’t any food back there,” Miller said. “Tara had her trunk in one of the holes, just listening to the sound. That let us know they do like it, and they’re very curious about it.”

Miller said because elephants have sharp memories and acute senses of hearing and smell, their habitats must be designed with that in mind.

Zoo Atlanta’s African Savanna elephant habitat was redesigned in 2019. In addition to the enrichment wall, it includes a bathing pond, two waterfalls, and swing boom devices that hold hay for elephants to eat as they would in the wild.

Miller said elephants sheltered at any zoo or conservation would benefit from enrichment devices enhanced by technology.

“I think anything they can participate in that gives them choice and control is great for all zoo elephants,” she said. “It depends on the elephants, but with our elephants, they can hear much higher frequencies than we can. That noise isn’t that loud for us, but for them, they’re feeling that noise, and they can hear much more, which makes it more stimulating for them.”

That support, however, begins to disappear in these restrictive cultures once women reach menopause, according to new research from Georgia Tech

Naveena Karusala, an assistant professor in Georgia Tech’s School of Interactive Computing, and master’s student Umme Ammara are working toward improving existing technologies and designing new ones for a demographic they believe has been neglected.

Karusala and Ammara co-authored a paper based on a study they conducted with women in urban Pakistan experiencing menopause.

“Women’s health is understudied in general, but menopause is more neglected than other women’s health issues,” Karusala said. “Our choice to focus on menopause is motivated by expanding how we holistically think about women’s well-being across their lifespan.”

Karusala and Ammara will present their paper in April at the 2026 ACM Conference on Human Factors in Computing Systems (CHI) in Barcelona.

Masking Symptoms

Menopause is diagnosed after 12 consecutive months without a period, vaginal bleeding, or spotting. The transition to menopause, called perimenopause, usually happens over two to eight years.

Hormone changes may cause symptoms such as irregular periods, vaginal dryness, hot flashes, night sweats, trouble sleeping, mood swings, and brain fog.

These symptoms can be debilitating in some cases and affect daily life. However, Ammara said women are pressured to remain silent, maintain appearances, and regulate their emotions to meet social expectations.

“Understanding menopause is important because a woman would be experiencing all these symptoms, and people will not understand those as actual symptoms,” Ammara said. “There’s been resistance to the idea of the medicalization of menopause. People don’t view it as an illness, but as a life transition and something that happens naturally.”

Feeling Isolated

The women interviewed by Karusala and Ammara either stayed at home full-time or were part of the workforce.

The researchers discovered that trusted family members might be the only sources women who stay at home and do not work turn to for disclosure. 

“Women at home have the flexibility to take breaks or work at their own pace, so a lot of their experience is shaped by the emotional barriers they face,” Ammara said. 

“That could come from their husbands and family members. Some are supportive and some are not. They might weaponize it and use that term against them, or they might dismiss what they’re going through.”

Ammara said it might be easier for women in the workforce to confide in their coworkers, but explaining to an employer that they need sick leave for menopause symptoms can be intimidating.

Even in online communities that have enabled women to anonymously share their health experiences, menopause is seldom discussed.

Raising Awareness

Karusala and Ammara argue in their paper that a public health approach could be the most effective way to spark conversation about menopause in a patriarchal culture in which technology use varies.

They said the challenge in implementing technologies geared toward menopause support is that the condition isn’t well understood in public. Improving maternal health, for example, is easier to promote within these societies because of the general understanding that motherhood is important.

“There must be an existing infrastructure to build on,” Karusala said. “For example, menstrual and maternal health are taught in schools and regularly discussed in primary care. Cultural and social meaning and importance are placed on motherhood.

“A lot of that doesn’t exist for menopause. Primary care doctors are unprepared to talk about menopause compared to other health issues.”

Design Solutions

Ammara said that the most effective way for technologies to make an impact on women going through menopause is to directly address systemic power structures around women’s health within Pakistani culture.

It can start with the husbands. 

“Framing the issue for husbands to understand menopause should be at the forefront of designing technology solutions,” she said. 

“In Islamic contexts, we suggest using faith-based framings. This has been proposed for maternal health in prior works that draw on Islamic principles to engage expectant fathers in providing care and support. Framing it around religious responsibility to involve men in the journey can also be done for menopause.”

A new mobile app will soon put the ability to monitor a baby’s prenatal heartbeat in the hands of pregnant women who may worry about their baby’s health in between doctor’s visits. 

Indoor farms, also known as vertical farms, are popular among agricultural researchers and are expanding across the agricultural industry. Some benefits they have over outdoor farms include:

• Year-round production of food crops
• Less water and land requirements
• Not needing pesticides
• Reducing carbon emissions from shipping
• Reducing food waste

Additionally, some studies indicate that indoor farms produce more nutritious food for urban communities. 

However, these farms are often inaccessible to birds, bees, and other natural pollinators, leaving the pollination process to humans. The tedious process must be completed by hand for each flower to ensure the indoor crop flourishes.

Ai-Ping Hu, a principal research engineer at the Georgia Tech Research Institute (GTRI), has spent years exploring methods to efficiently pollinate flowering plants and food crops in indoor farms to find a way to efficiently pollinate flower plants and food crops in indoor farms.

Hu, Assistant Professor Shreyas Kousik of the George W. Woodruff School of Mechanical Engineering, and a rotating group of student interns have developed a robot prototype that may be up to the task.

The robot can efficiently pollinate plants that have both male and female reproductive parts. These plants only require pollen to be transferred from one part to the other rather than externally from another flower.

Natural pollinators perform this task outdoors, but Hu said indoor farmers often use a paintbrush or electric tootbrush to ensure these flowers are pollinated. 

Knowing the Pose

An early challenge the research team addressed was teaching the robot to identify the “pose” of each flower. Pose refers to a flower’s orientation, shape, and symmetry. Knowing these details ensures precise delivery of the pollen to maximize reproductive success. 

“It’s crucial to know exactly which way the flowers are facing,” Hu said.

“You want to approach the flower from the front because that’s where all the biological structures are. Knowing the pose tells you where the stem is. Our device grasps the stem and shakes it to dislodge the pollen.

“Every flower is going to have its own pose, and you need to know what that is within at least 10 degrees.”

Computer Vision Breakthrough

Harsh Muriki is a robotics master’s student at Georgia Tech’s School of Interactive Computing, who used computer vision to solve the pose problem while interning for Hu and GTRI.

Muriki attached a camera to a FarmBot to capture images of strawberry plants from dozens of angles in a small garden in front of Georgia Tech’s Food Processing Technology Building. The FarmBot is an XYZ-axis robot that waters and sprays pesticides on outdoor gardens, though it is not capable of pollination.

“We reconstruct the images of the flower into a 3D model and use a technique that converts the 3D model into multiple 2D images with depth information,” Muriki said. “This enables us to send them to object detectors.”

Muriki said he used a real-time object detection system called YOLO (You Only Look Once) to classify objects. YOLO is known for identifying and classifying objects in a single pass.

Ved Sengupta, a computer engineering major who interned with Muriki, fine-tuned the algorithms that converted 3D images into 2D.

“This was a crucial part of making robot pollination possible,” Sengupta said. “There is a big gap between 3D and 2D image processing.

“There’s not a lot of data on the internet for 3D object detection, but there’s a ton for 2D. We were able to get great results from the converted images, and I think any sector of technology can take advantage of that.”

Sengupta, Muriki, and Hu co-authored a paper about their work that was accepted to the 2025 International Conference on Robotics and Automation (ICRA) in Atlanta.

Measuring Success

The pollination robot, built in Kousik’s Safe Robotics Lab, is now in the prototype phase. 

Hu said the robot can do more than pollinate. It can also analyze each flower to determine how well it was pollinated and whether the chances for reproduction are high.

“It has an additional capability of microscopic inspection,” Hu said. “It’s the first device we know of that provides visual feedback on how well a flower was pollinated.”

For more information about the robot, visit the Safe Robotics Lab project page.

Georgia Stoica is one of 38 Ph.D. students worldwide researching machine learning who were named a 2025 Google Ph.D. Fellow.

Stoica is designing AI training methods that bypass fine-tuning, which is the process of adapting a large pre-trained model to perform new tasks. Fine-tuning is one of the most common ways engineers update large-language models like ChatGPT, Gemini, and Claude to add new capabilities. 

If an AI company wants to give a model a new capability, it could create a new model from scratch for that specific purpose. However, if the model already has relevant training and knowledge of the new task, fine-tuning is cheaper.

Stoica argues that fine-tuning still uses large amounts of data, and that other methods can help models learn more effectively and efficiently.

“Full fine-tuning yields strong performance, but it can be costly, and it risks catastrophic forgetting,” Stoica said. “My research asks if we can extend a model’s capabilities by imbuing it with the expertise of others, without fine-tuning?

“Reducing cost and improving efficiency is more important than ever. We have so many publicly available models that have been trained to solve a variety of tasks. It’s redundant to train a new model from scratch. It’s much more efficient to leverage the information that already exists to get a model up to speed.”

Stoica said the solution is a cost-effective method called model merging. This method combines two or more AI models into a single model, improving performance without fine-tuning.

On a basic level, Stoica said an example would be combining a model that is efficient at classifying cats with one that works well at dogs.

“Merging is cheap because you just take the parameters, the weights of your existing models, and combine them,” he said. “You could take the average of the weights to create a new model, but that sometimes doesn’t work. My work has aimed to rearrange the weights so they can communicate easily with each other.”

Through his Google fellowship, Stoica seeks to apply model merging to create a cutting-edge vision encoder. A vision encoder converts image or video data into numerical representations that computers can understand. This enables tasks such as image or facial recognition and generative image captioning.

“I want to be at the frontier of the field, and Google is clearly part of that,” Stoica said. “The vision encoder is very large-scale, and Google has the infrastructure to accommodate it.”

Associate Professor Jessica Roberts and Ph.D. student Emily Amspoker of Georgia Tech’s School of Interactive Computing are working with the University of Georgia’s Marine Extension and Georgia Sea Grant in Savannah. Together, they’ve developed a prototype display that uses sonification and texture to convey sea floor habitat information from Gray’s Reef National Marine Sanctuary off the coast of Georgia.

Sonification is the process of translating data points into sound.

The display functions as a map that BLV users can follow to learn about each habitat. It is made from a wooden board with laser-cut patterns engraved into the surface. Each pattern represents information about the four types of habitats found in Gray’s Reef. Each pattern has a distinct sound that corresponds to a legend on the board, which provides an audio description of each habitat.

The four habitats are:

• Flat sand — smooth sandy seafloor with little topographic variation that provides habitat for burrowing organisms such as worms, clams, and sand dollars.
• Rippled sand — sandy bottom shaped into small wave-like ridges by currents and wave action; supports microhabitats of small invertebrates and attracts fish feeding on buried prey.
• Sparse live bottom — areas of exposed hard surfaces with scattered attached organisms like sponges, corals, and algae, offering structure and shelter for reef-associated fish and invertebrates.
• Dense live bottom — hard-bottom reef areas with abundant attached marine life, providing high biodiversity and offering food, and breeding sites for numerous species.

By allowing learners to explore these habitats, the team hopes to emphasize the importance of protecting diverse ocean habitats. 

“Our job was to figure out how we can use sounds and touch to represent each of the four habitat types so our visitors can explore the ocean without being able to see it,” she said.

Roberts said the project is critical to advance understanding of how science and informal learning can be more inclusive to those who have difficulty processing visual data displays.

Ryan Punamiya (CS 2025) and Summer Abramson, a third-year computational media student, have been honored by the Computing Research Association (CRA) through its 2025–2026 Outstanding Undergraduate Researcher Award (URA) program. 

Punamiya was named a runner-up for the prestigious award, while Abramson received an honorable mention among hundreds of applicants from universities across North America. 

The CRA Outstanding Undergraduate Researcher Award program recognized eight awardees in 2026, along with eight runners-up, nine finalists, and over 200 honorable mentions from thousands of applications.  

Advancing Robotics Research 

Punamiya knew early on that he didn’t want to wait until starting his Ph.D. to do meaningful and impactful robotics research.  

Punamiya joined the Robot Learning and Reasoning Lab (RL2) directed by Assistant Professor Danfei Xu. While there, he contributed to the lab’s Meta-sponsored EgoMimic project, which trains robots to perform human tasks using recordings captured by Meta’s Project Aria research glasses. 

Punamiya is also the first author of a paper accepted to the 2025 Conference on Neural Information Processing Systems (NeurIPS), one of the world’s most prestigious artificial intelligence (AI) and machine learning conferences. 

“Ryan is the strongest undergraduate I've worked with,” Xu said, “including students who went on to Stanford, Berkeley, and leadership roles in major tech companies. He’s already operating at the level of a strong third-year Ph.D. student.” 

Punamiya said it was a challenge to balance his undergraduate coursework with his research in Xu’s lab. 

“You get out how much you put in,” he said. “I built my class schedule to give myself as much time to do research as possible. It also boils down to having the right research mentors. 

“(Xu) never saw me as an undergrad who’s just there to do grunt work. I was fortunate he saw my curiosity and cultivated me as a researcher. That’s really how you get more undergrads motivated to research — giving them the chance to be independent and explore ideas of their own.” 

Punamiya said his work in Xu’s lab has already helped him identify the research areas he wants to focus on as he considers his next steps. He will continue developing generalized training models for robots using human data so they can perform tasks instantly upon deployment. 

"The amount of data needed to train a robot is difficult to obtain even for top industry companies," he said. "We have embodied robot data available in billions of humans. With the advent of extended reality devices, we can get a scalable source of diverse interactions within environments."

Punamiya graduated in December and recently started an internship at Nvidia. He mentioned he has been accepted into several Ph.D. programs, including Georgia Tech, and he is choosing where to continue his research. 

“It’s the first time my research has been acknowledged externally by the robotics community,” he said. “It’s good to know the problem I’m working on is important, and that motivates me. Robotics is an exciting field. We are doing things now that two years ago were difficult to do.” 

Researching Inclusion in Computing Education 

Abramson conducts research in the People-Agents Research for Computing Education (PARCE) Laboratory under the mentorship of Pedro Guillermo Feijóo-García, a faculty member in the School of Computing Instruction. He and the Associate Dean for Undergraduate Education, Olufisayo Omojokun, nominated her for the award. 

Her work focuses on the intersection of computing education and human-AI interaction, where she’s been exploring ways to create more equitable technology. 

“This is such a huge milestone, and I couldn't be prouder of Summer,” Feijóo-García said. “Mentoring her for almost two years has been an amazing experience.” 

Abramson has received the Georgia Tech President’s Undergraduate Research Award (PURA) twice, which supports her research exploring how user-centered design curricula can help address attrition among women in computing.

“I’ve had the amazing opportunity to pursue research at the intersection of student identity, community belonging, and how we can build tools that support our diverse student population,” Abramson said. 

“Dr. Pedro and I have a goal to build community through a human-first approach, and I could not be more grateful for his support and guidance in my own journey. The CRA highlights the best of what the computing discipline has to offer, and I am incredibly honored for our work to be recognized.”

Abramson will spend the summer researching how user-centered design curricula can help promote confidence, belonging, and retention for women in computing.

Nominees for the PURA program were recognized for contributing to multiple research projects, authoring or coauthoring papers, presenting at conferences, developing widely used software artifacts, and supporting their communities as teaching assistants, tutors, and mentors. 

School of Computing Instruction Communications Officer Emily Smith contributed to this story.

Main Photo: Ryan Punamiya works with a robot during the 2025 International Conference on Robotics and Automation in Atlanta. Photo by Terence Rushin/College of Computing.

Jennifer Kim and Mark Riedl recently received a $500,000 NSF grant to develop large language models (LLMs) that provide strength-based job coaching for autistic job seekers. 

The two Georgia Tech researchers work with Heather Dicks, a career development advisor in Georgia Tech’s EXCEL program, and other nonprofit organizations to provide job-seeking resources to autistic people.

Dicks said the average job search for people with autism can take three to six months in a good economy. It can take up to 18 months in a bad one. However, the new LLMs from Georgia Tech could help to reduce stress and fast-track these job seekers into employment.

Kim is an assistant professor who specializes in human-computer interaction technology that benefits neurodivergent people. Riedl is a professor and an expert in the development of artificial intelligence (AI) and machine learning technologies.

The team’s goal is to identify job-search pain points and understand how job coaches create better employment prospects for their autistic clients.

“Large-language models have an opportunity to support this kind of work if we can have more data about each different individual strength,” Kim said.

“We want to know what worked for them in specific settings at work, what didn’t work, and what kind of accommodations can better help them. That includes how they should prepare for interviews, how they can better represent their skills, how they can address accommodations they need, and how to write a cover letter. It’s a broad range.”

Dicks has advocated for neurodivergent people and helped them find employment for 20 years. She worked at the Center for the Visually Impaired in Atlanta before coming to Georgia Tech in 2017.

She said most nonprofits that support neurodivergent people offer career development programs and many contract job coaches, but limited coach availability often leads to long waitlists. However, LLMs could fill this availability gap to address the immediate needs of job seekers who may not have access to a job coach.

“These organizations often run at a slow pace, and there’s high turnover,” Dicks said. “An AI tool could get the job seeker quicker support. Maybe they don’t even need to wait on the government system.

“If they’re on a waitlist, it can help the user put together a resume and practice general interview questions. When the job coach is ready to work with them, they’re able to hit the ground running.”

Nailing the Interview

Dicks said the job interview is one of the biggest challenges for people with autism.

“They have trouble picking up on visual and nonverbal cues — the tone of the interview, figuring out the nuances that a question is hinting at,” she said. “They’re not giving the warm and fuzzy vibes that allow them to connect on a personal level.”

That’s why Kim wants the models to reflect a strength-based coaching approach. Strength-based coaching is particularly effective for individuals with autism. Many possess traits that employers value. These include:

• Close attention to detail
• Strong technical proficiency
• Unique problem-solving perspectives

“The issue is that they don’t know how these strengths can be applied in the workplace,” Kim said. “Once they understand this, they can communicate with employers about their strengths and the accommodations employers should provide to the job seeker so they can successfully apply their skills at work.”

Handling Rejection

Still, Kim understands that candidates will need to handle rejection to make it through the search process. She envisions LLMs that help them refocus their energy and regain their confidence after being turned down.

“When you get a lot of rejection emails, it’s easy to feel you’re not good enough,” she said. “Being constantly reminded about your strengths and their prior successes can get them through the stressful job-seeking process.”

Dicks said the models should also be able to provide feedback so that candidates don’t repeat mistakes.

“It can tell them what would’ve been a better answer or a better way to say it,” Dicks said. “It can also encourage them with reminders that you get 100 noes before you get a yes.”

You’re Hired, Now What?

Dicks said the role of a job coach doesn’t end the moment a client is hired. Government-contracted job coaches may work with their clients for up to 90 days after they start a new job to support their transition.

However, she said, sometimes that isn’t enough. Many companies have probationary periods exceeding three months. Autistic individuals may struggle with on-the-job training or communicating what accommodations they need from their new employer. 

These are just a few gaps an AI tool can fill for these individuals after they’re hired.

“I could see these models evolving to being supportive at those critical junctures of the probationary period being over or the one-year job review or the annual evaluation that everyone dreads,” she said.

Dicks has an average caseload of 15 students, whom she assists in landing jobs and internships through the EXCEL program.

EXCEL provides a mentorship program for students with intellectual and developmental disabilities from the time they set foot on campus through graduation and beyond.

For more information and to apply, visit EXCEL’s website.

However, most popular AI-powered chatbots are not trained to recognize when someone is in crisis. LLMs also cannot determine when to refer someone to a human specialist.

New Georgia Tech research projects that address these issues may soon provide people seeking mental health support with safer experiences. 

Google has awarded research grants to three faculty members from the School of Interactive Computing to study artificial intelligence (AI), trust, safety, and security. The grants were among dozens awarded by the company to researchers across the country.

Professor Munmun De Choudhury, Associate Professor Rosa Arriaga, and Associate Professor Alan Ritter are among the recipients of the 2025 Google Academic Research Awards. 

Their projects will explore questions like:

• What harms could occur if people consult LLMs for mental health advice?
• Which groups are most at risk of receiving harmful guidance?
• When should an LLM stop responding and refer someone to a human professional?

De Choudhury and Arriaga will examine how LLMs might harm people seeking mental health care.

De Choudhury’s work focuses on spotting when chatbot conversations go wrong and lead users toward self-harm. She is also studying design changes that could prevent these situations.

Her project, Exiting Harmful Reliance: Identifying Crises & Care Escalation Needs, is in partnership with Angel Hsing-Chi Hwang from the University of Southern California. Together, they will review real and synthetic chat transcripts with clinicians to find language patterns that signal risk.

“A chatbot will always give a response and keep talking to you for however long you want,” De Choudhury said. “That may not be a good thing for someone in crisis. We need to know when the right response is to stop and suggest talking to a human.”

Understanding Risks for Low-Income Users

Arriaga’s project, Dull, Dirty, Dangerous: Investigating Trust of Digital Resources Among Low-SES Mental Health Care Seekers, looks at how LLMs affect people with low socioeconomic status (SES).

Dull, dirty, and dangerous is a phrase used to describe work that is well-suited for robot automation because they are repetitive, physically taxing, or hazardous for humans. Arriaga said she adapted these terms for her research to create a taxonomy of the harms AI can cause to people seeking mental health care.

Arriaga also wants to label the trust factors that chatbots have that attract low-SES users to seek their advice, and how these may differ for adults and adolescents across contexts. 

“We know one of the reasons some users go to LLMs is because they aren’t insured and can’t afford a therapist,” she said. “LLMs are available 24-7. Maybe it doesn’t start as a trust issue. Maybe it starts with availability. 

“Some of these human-AI conversations that result in harmful mental health advice didn’t begin on the topic of mental health. In one case, the person started going to the machine for help with homework.

“Then this relationship evolved into personal matters. Should we constrain the system to limit itself to helping someone with their homework and not wander off that subject into mental health matters?”

Managing Privacy Risks for Social Media

Ritter will use the Google award to advance research on social media privacy tools, including interactive AI agents that help people make more informed decisions about what they share online.

His project, AI Tools to Help Users Make Informed Decisions About Online Information Sharing, focuses on reducing privacy risks in both text and images by identifying when posts reveal more than users intend.

“We’ve been developing methods to assess risks in text, and now we’re extending that work to images,” Ritter said. “People post photos without realizing how easily they can be geolocated by advanced AI systems. A casual selfie near home might contain subtle cues about where you live, like a street sign, that reveal private details.”

The project aims to create AI agents that review content within user posts, flag elements that pose risk, and suggest safer alternatives. Ritter said he wants people to maintain control over their privacy without limiting freedom of expression.

Ritter will deploy advanced reasoning models capable of probabilistic privacy estimation. These systems can infer how identifiable a piece of text might be or how likely an image is to reveal a user’s location.

For images, Ritter and his collaborators will use models that identify geolocatable features, allowing users to edit or hide them before posting.

For more on Ritter’s research, read how an LLM he co-developed protects the privacy of users on social media.

Some were still alive, but most of the birds were dead. They were all too easy to find.

“I knew birds hit buildings, but I didn’t know much more about the issue at that time, and I was surprised how easily I just found birds,” Betuel said.

Birds flying into windows aren’t isolated events. Environmentalists estimate between 365 million and one billion birds die each year from colliding with structures in the U.S.  

“That statistic is hard for most people to comprehend,” Betuel said. “When you think about the millions of homes we have and these high-rise buildings, and if each one is killing a few a year, that number can get big pretty quick.”

Betuel is the executive director of Birds Georgia, a nonprofit affiliate of the Audubon network that leads bird conservation efforts in Georgia. For 10 years, volunteers from the organization have combed Atlanta’s streets, collecting bird specimens.

Birds Georgia launched Project Safe Flight in 2015 to reduce bird building-collision mortality through data collection. Through legislation, the group aims to make building construction bird-friendly and reduce light pollution.

Environmentalists who study the issue have ranked Atlanta, which sits squarely on a migration route, as the fourth-most dangerous city for birds during fall migration. It is the ninth-most dangerous city during spring migration.

The number of bird deaths from collisions in Atlanta and across the state remains unknown. However, new data tools developed by student researchers in the College of Computing at Georgia Tech are helping Birds Georgia get a clearer picture of the issue.

“We’ve been working with different folks at Georgia Tech for years now, but it’s really picked up lately,” Betuel said. “There’s a lot of momentum and interest on campus to try to make the city safer for birds.”

Pushing for Policy

Ashley Boone, a Ph.D. student in human-centered computing in Tech’s School of Interactive Computing, has led the student effort to help Birds Georgia organize its data. 

Boone said organizing data and knowing how to use it is critical to spark conversations about adopting legislation.

“We often see a gap between data collection and data advocacy,” she said. “Birds Georgia has done an amazing job of tracking collisions in Atlanta over the last 10 years. My goal is to understand the role technology can play in making data useful for policy change.”

User-interface tools designed by computer science undergraduate students James Kemerait and Ian Wood have ramped up that process. One tool converts data input into visualizations optimized for social media, while another consolidates the data collected by volunteers and external sources.

Boone said the desired legislation would mirror policies implemented by New York City. Those policies require the use of bird-safe materials — like window film with patterned designs that break up reflections — in new buildings and buildings undergoing significant renovations. 

What Can Residents Do?

Residents, whose homes account for about 40% of bird collision deaths in the U.S., can also make an impact.

“Households are an underexamined cause of bird collisions,” Boone said. “We focus on the big buildings because it’s easier to convince one manager of a large building to use bird-safe materials, and it’s easier for a policy to address a commercial building. But the sheer volume of residential buildings in the U.S. has a tremendous impact on the number of collisions.”

Steps that homeowners can take include:

• Buying bird-safe film or making do-it-yourself versions of it to put on windows.
• Placing attractive objects like birdhouses and birdfeeders very close or very far away from windows.
• Turning off lights after 9 p.m. on the busiest migration nights of the year.

Betuel said millions of birds can fly over Atlanta on a single night during migration, and they are attracted to the city lights.

“They’ll come into urban centers and collide with an illuminated building, or maybe they overnight somewhere that isn’t safe,” he said. “The next day, they’re surrounded by glass, and birds don’t understand reflection.”

Residents can visit the Birds Georgia website to sign up for the Lights Out Pledge. Those who sign up will receive a text on the 10 busiest migratory nights of the year, and they will be asked to turn their lights off early.

The tools provided by Georgia Tech gave Birds Georgia insight into the number of bird species affected by collisions — more than 140, according to Betuel.

Betuel said that when the organization reaches an estimate of bird collisions, he hopes the number will raise alarms and turn people’s attention to the ecological impact. 

“All these birds being lost results in fewer birds to eat pest insects, fewer birds to pollinate flowers, fewer birds to disperse seeds — all the ecological functions that we need, that they’re doing in the background that most people aren’t keen to,” he said. “If this decline in bird life continues to happen, at some point, there will be issues with our ecosystems functioning as they always have.”

That’s usually the only information Brandon Biggs receives from digital maps.

Biggs is a human-centered computing Ph.D. student in Georgia Tech’s School of Interactive Computing. He is almost totally blind due to Leber’s Congenital Amaurosis (LCA), a rare degenerative eye disorder affecting about one in 40,000 people.

Based on his experience, Biggs argues that most digital maps aren’t accessible to people who are blind. Even worse, he said, the needs of the blind are usually overlooked.

“When I started research on maps, I had never viewed a weather, campus, or building map, so I didn’t realize the amount of information maps contain,” Biggs said. “How do you represent shapes, orientation, and layout through audio and translate that into a geographic map?”

To answer these questions, Biggs founded XRNavigation, a company focused on developing accessible digital tools. Its flagship product, Audiom, is a cross-sensory map that people can see and hear through text.

“Sighted people view about 300 maps per year, while blind people view fewer than one,” he said. “Blind people don’t view maps; it’s not part of their lives.

“I want to ensure that for blind users, digital maps are no longer just ‘blank.’  They receive the information they need to know to navigate in this world and become more autonomous.”

Organizations that need to include accessible maps in their digital spaces can integrate Audiom into their website or app. 

Georgia Tech recently became one such organization and used Audiom to introduce the first fully accessible digital campus map.

Professor Bruce Walker advises Biggs in Walker’s Sonification Lab, which designs auditory displays for technologies.

“Brandon has the perfect and unique blend of technical skills, research savvy, innovativeness, lived experience, and never-stop attitude to tackle this problem while impacting and improving many lives,” Walker said.

Defining Accessibility

Biggs said most maps limit accessibility features to turn-by-turn directions, tables, or other kinds of alternative text that disregard spatial information. The ability to communicate spatial information distinguishes Audiom.

“According to Web Content Accessibility Guidelines (WCAG), all non-text content — like maps — must include a text alternative with an equivalent purpose,” Biggs said. “But what does ‘equivalent purpose’ mean for geographic maps?

“We argue that every single map, regardless of what it’s showing, communicates general spatialized information and relationships.”

Audiom also prioritizes the information that’s most important to blind users, including sidewalks and buildings.

“There’s a lot of information blind people just don’t get on maps but desperately need,” he said. “They couldn’t care less about the roads. They might need the road name, but they really need the sidewalks.

“If a blind person made a map, they might not even add the roads. And then they would add in the location of doorways, a critical detail that sighted people completely leave out.”

Biggs’s work is already gaining national recognition. XRNavigation was recently one of three companies selected by the Global Accessibility Awareness Day (GAAD) Foundation for a 2025 Gaady Award, which honors work being done to make digital technologies more accessible.

Past and present winners of Gaady Awards range from tech startups to major brands like T-Mobile.

Biggs will accept the award during a banquet on Thursday in San Francisco.

311 systems allow residents to report potholes, broken fire hydrants, and other municipal issues. In recent years, the use of artificial intelligence (AI) to provide 311 services to community residents has boomed across city and state governments. This includes an artificial virtual assistant (AVA) developed by third-party vendors for the City of Atlanta in 2023.

Through survey data, researchers from Tech’s School of Interactive Computing found that many residents are generally positive about 311 chatbots. In addition to eliminating long wait times over the phone, they also offer residents quick answers to permit applications, waste collection, and other frequently asked questions.

However, the study, which was conducted in Atlanta, indicates that 311 chatbots could be causing residents to feel isolated from public officials and less aware of what’s happening in their community.

Jieyu Zhou, a Ph.D. student in the School of IC, said it doesn’t have to be that way.

Uniting Communities

Zhou and her advisor, Assistant Professor Christopher MacLellan, published a paper at the 2025 ACM Designing Interactive Systems (DIS) Conference that focuses on improving public service chatbot design and amplifying their civic impact. They collaborated with Professor Carl DiSalvo, Associate Professor Lynn Dombrowski, and graduate students Rui Shen and Yue You.

Zhou said 311 chatbots have the potential to be agents that drive community organization and improve quality of life.

“Current chatbots risk isolating users in their own experience,” Zhou said. “In the 311 system, people tend to report their own individual issues but lose a sense of what is happening in their broader community. 

“People are very positive about these tools, but I think there’s an opportunity as we envision what civic chatbots could be. It’s important for us to emphasize that social element — engaging people within the community and connecting them with government representatives, community organizers, and other community members.”

Zhou and MacLellan said 311 chatbots can leave users wondering if others in their communities share their concerns.

“If people are at a town hall meeting, they can get a sense of whether the problems they are experiencing are shared by others,” Zhou said. “We can’t do that with a chatbot. It’s like an isolated room, and we’re trying to open the doors and the windows.”

Adding a Human Touch

In their paper, the researchers note that one of the biggest criticisms of 311 chatbots is they can’t replace interpersonal interaction.

Unlike chatbots, people working in local government offices are likely to:

• Have direct knowledge of issues
• Provide appropriate referrals
• Empathize with the resident’s concerns

MacLellan said residents are likely to grow frustrated with a chatbot when reporting issues that require this level of contextual knowledge.

One person in the researchers’ survey noted that the chatbot they used didn’t understand that their report was about a sidewalk issue, not a street issue.

“Explaining such a situation to a human representative is straightforward,” MacLellan said. “However, when the issue being raised does not fall within any of the categories the chatbot is built to address, it often misinterprets the query and offers information that isn’t helpful.”

The researchers offer some design suggestions that can help chatbots foster community engagement and improve community well-being:

• Escalation. Regarding the sidewalk report, the chatbot did not offer a way to escalate the query to a human who could resolve it. Zhou said that this is a feature that chatbots should have but often lack.
• Transparency. Chatbots could provide details about recent and frequently reported community issues. They should inform users early in the call process about known problems to help avoid an overload of user complaints.
• Education. Chatbots can keep users updated about what’s happening in their communities.
• Collective action. Chatbots can help communities organize and gather ideas to address challenges and solve problems.

“Government agencies may focus mainly on fixing individual issues,” Zhou said, “But recognizing community-level patterns can inspire collective creativity. For example, one participant suggested that if many people report a broken swing at a playground, it could spark an initiative to design a new playground together—going far beyond just fixing it.”

These are just a few examples of things, the researchers argue, that 311 services were originally designed to achieve.

“Communities were already collaborating on identifying and reporting issues,” Zhou said. “These chatbots should reflect the original intentions and collaboration practices of the communities they serve.

“Our research suggests we can increase the positive impact of civic chatbots by including social aspects within the design of the system, connecting people, and building a community view.”

At its event this week in Alberta, Canada, the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE) is honoring Professor Mark Riedl. The award also honors University of Utah Professor and Division of Games Chair Michael Young, Riedl’s Ph.D. advisor.

Riedl studied under Young at North Carolina State University.

Their 2005 paper, From Linear Story Generation to Branching Story Graphs, highlighted the challenges of using AI to create interactive gaming narratives in which user actions influence the story’s progression. 

In 2005, computer game systems that supported linear, non-branching games were widely used. Riedl introduced an innovative mathematical formula for interactive stories ranging from choose-your-own-adventure novels to modern computer games.

“We didn’t use the term ‘generative AI’ back then, but I was working on AI for the generation of creative artifacts,” Riedl said. “This was before we had practical deep learning or large language models.

“One of the reasons this paper is still relevant 20 years later is that it didn’t just present a technology, it attempted to provide a framework for solving a grand challenge in AI.”

That challenge is still ongoing, Riedl said. Game designers continue to struggle with balancing story coherence against the amount of narrative control afforded to users.

“When users exercise a high degree of control within the environment, it is likely that their actions will change the state of the world in ways that may interfere with the causal dependencies between actions as intended within a storyline,” Riedl and Young wrote in the paper.

“Narrative mediation makes linear narratives interactive. The question is: Is the expressive power of narrative mediation at least as powerful as the story graph representation?”

AIIDE is being held this week at the University of Alberta in Edmonton, Alberta. Riedl will receive the award on Wednesday.

World foundation models (WFMs) enable physical AI systems to learn and operate within synthetic worlds created by generative artificial intelligence (genAI). For example, these models use predictive capabilities to generate up to 30 seconds of video that accurately reflects the real world.

The new framework, developed by a Georgia Tech researcher, enhances the processing speed of the neural networks that simulate these real-world environments from text, images, or video inputs.

The neural networks that make up the architectures of large language models like ChatGPT and visual models like Sora process contextual information using the “attention mechanism.”

Attention refers to a model’s ability to focus on the most relevant parts of input.

The Neighborhood Attention Extension (NATTEN) allows models that require GPUs or high-performance computing systems to process information and generate outputs more efficiently.

Processing speeds can increase by up to 2.6 times, said Ali Hassani, a Ph.D. student in the School of Interactive Computing and the creator of NATTEN. Hassani is advised by Associate Professor Humphrey Shi.

Hassani is also a research scientist at Nvidia, where he introduced NATTEN to Cosmos — a family of WFMs the company uses to train robots, autonomous vehicles, and other physical AI applications.

“You can map just about anything from a prompt or an image or any combination of frames from an existing video to predict future videos,” Hassani said. “Instead of generating words with an LLM, you’re generating a world.

“Unlike LLMs that generate a single token at a time, these models are compute-heavy. They generate many images — often hundreds of frames at a time — so the models put a lot of work on the GPU. NATTEN lets us decrease some of that work and proportionately accelerate the model.”

While some of those users can upgrade to Windows 11, hundreds of millions of devices don’t meet the technical requirements. 

Those users might be wondering what else they can do besides throwing away their current device and buying a new one or risking running outdated software on it.

The tech conglomerate faced backlash from environmental and cybersecurity experts after informing Windows users that it would cease providing updates for Windows 10. 

These experts have warned that rendering hundreds of millions of devices practically useless will worsen the ever-growing problem with electronic waste (e-waste) and leave users who can't upgrade vulnerable to cybersecurity threats.

Researchers from Georgia Tech’s School of Interactive Computing (SIC) and School of Cybersecurity and Privacy (SCP) echo those concerns.

Forcing users to replace their devices means that up to 240 million old devices, according to one analysis, will inevitably end up in landfills.

“The problem of e-waste raises the question of why and how these technologies become obsolete,” said Cindy Lin, a Stephen Fleming Early Career Assistant Professor in SIC. 

Lin studies data structures and environmental governance in Southeast Asia and the U.S.

“Scholarship in human-computer interaction (HCI) on repair reveals that many of these technologies suffer from planned obsolescence,” she said. “This means that companies have designed products with a short lifespan, increasing consumption and waste simultaneously.”

When e-waste is dumped in landfills, the organic materials within devices decompose, producing methane, a potent greenhouse gas. And with every discarded device comes the need to produce new ones. The raw materials of these devices are mined, refined, and processed, consuming enormous amounts of energy through the burning of fossil fuels.

The Problem With Hackers

Though Microsoft said it will continue to provide Windows 10 security updates for one year, users are still being pressured to upgrade. By this time next year, if users still haven’t upgraded to Windows 11, they can expect to become easy targets for cyber criminals.

For example, users could receive phishing emails claiming to be from Microsoft about security updates from hackers pretending to be Microsoft. 

“The cybersecurity implications are very serious because new vulnerabilities of Windows 10 will go unpatched for a large part of the user base of this system,” said Mustaque Ahamad, Regents’ Entrepreneur Professor and interim chair of SCP.

“These users will become targets of hackers and cyber criminals who will be able to exploit these vulnerabilities. This will make these machines more prone to attacks such as ransomware and data exfiltration.”

What Can Users Do?

Buying a new device typically costs around $300 at the low end, while some gaming computers can exceed $2,500. 

Josiah Hester, an associate professor in the School of IC who researches computing and sustainability, said users who want to avoid discarding their devices can install Linux Mint, a free universal operating system.

“I would hope that instead of discarding, people might see this as an opportunity to go into a more open ecosystem like Linux Mint, which was designed for Windows users,” Hester said. 

“So much perfectly good hardware is obsolesced by force, when users are more than willing to give it a second life, either through ending support on the software side, subscription services that require certain versions of an OS, or even building the hardware or low-level functions that reduce the autonomy of device owners.” 

Linux Mint is open source and offers its own suite of software products, including a word processor. It also has a built-in security system. It requires 2GB of RAM, 20GB of disk space, and 1024x768 resolution to operate.

On a systemic level, Lin and Hester said people can support organizations that advocate for right to repair and legislation that protects consumers from planned obsolescence.

“HCI studies of informal economies of improvisation and repair have demonstrated that technologies have a longer lifecycle if we have access to expertise on how to repair them without facing penalties such as copyright violations,” Lin said.

“The ongoing right-to-repair movement in the US shows promise in making technology repairable and, in turn, more sustainable.”

Stasko achieved a great deal during his 36 years as a professor at Georgia Tech and made significant contributions to data visualization research and innovations. He is a Fellow of the ACM and IEEE and received the IEEE Visualization and Graphics Technical Community Lifetime Achievement Award. 

In all those years, none of his students or colleagues could recall a moment when he didn’t demonstrate kindness.

“He supported me in fleshing out my ideas into a Ph.D. dissertation,” said Dean Jerding (CS Ph.D. 1997), one of Stasko’s former students. “He was always calm and communicated any criticism in a very positive way. He never said I had a dumb idea. He was always encouraging, and he redirected you with his input.”

The School of Interactive Computing bid farewell to Stasko on Thursday, following his official retirement in July. 

During the event, Shaowen Bardzell, School of IC chair and professor, announced the establishment of the John Stasko Award for Teaching Excellence in Stasko’s honor. Bardzell said the award will be given each year to as many as “two faculty members in the School of Interactive Computing whose teaching and mentoring channel John’s passion and care for our students.”

“You can be effective while being nice, and you can be heard while being quiet and thoughtful,” said Keith Edwards, a professor in the School of IC who was one of Stasko’s first students. “He’s the same even-keeled, thoughtful person as he was when I first knew him. He’s very generous. If it hadn’t been for John, I think there’s a chance I would’ve fallen through the cracks when I was looking for an advisor at Georgia Tech. I’m very fortunate he took me on.”

New College, New Blood

Stasko came to Georgia Tech in 1989 fresh off completing his Ph.D. in computer science at Brown University. That was a year before the establishment of the College of Computing at Georgia Tech. The computer science program was administered by the School of Information and Computer Science, which was housed in the College of Sciences.

“It was exciting because we were igniting computer science at Georgia Tech, and there were a lot of young faculty like me who were brand new, right out of college,” Stasko said. “There was this spirit of working together and wanting to make something great here.”

Stasko said when the College of Computing was established in 1990, Georgia Tech ranked outside the top 20 of U.S. News and World Report’s computer science program rankings. 

Many new faculty members like Stasko were interested in data visualization, computer graphics, and human-computer interaction. Georgia Tech quickly bolstered its computer science reputation by positioning itself at the forefront of those emerging fields with the creation of the Graphics, Visualization, and Usability (GVU) Center.

“A lot of the top five to 10 schools like Stanford, MIT, and Berkeley were very strong in the traditional subareas of computer science,” Stasko said. “I think it helped us to develop a strength in HCI, graphics and visualization. We were one of the earliest to embrace those, so it made it easier for us to shine. U.S. News and World Report had a new sub-ranking called Graphics and HCI, and we were ranked No. 1 very early on. That really helped us.”

Growing as a Mentor

Stasko credits Jim Foley, the first director of GVU, who now has a scholarship named in his honor for outstanding graduate students, as the model for how to conduct oneself as a teacher.

“Jim was the most wonderful mentor I could’ve had,” Stasko said. “He was a famous professor, and everyone in computer science around the country knew him, but he was always so humble, and he would meet all the new junior faculty and want to help us get going. He allowed us to shine.”

Stasko became most well-known for his research, particularly for his invention of Jigsaw in 2007. Jigsaw is a visualization algorithm that can create a visual index of a large document collection.

“It could help an analyst see the story that’s spread across 1,500 different documents about a police case, for example,” he said. “Or maybe they were reviews of a product that you wanted to learn about, or which car or which TV you should buy without having to read 1,500 reviews. We used early machine learning methods to analyze the text and created a suite of different visualizations communicating that analysis.”

In addition to his research, Stasko taught an intro to JavaScript course for 20 years to thousands of Tech students. Though it wasn’t required of him to teach it, he said he enjoyed interacting with incoming first-year students because it “helped keep me feeling young.”

In 2007, Stasko joined the faculty of the newly created School of Interactive Computing. He served as the interim chair of the school from 2021 to 2022, and he was also named Regents’ Professor by the University System of Georgia in 2021.

Leaving a Legacy

Today, the College of Computing has the No. 5 undergraduate and No. 6 graduate computer science program in the U.S. and is the largest college on Georgia Tech’s campus.

“I’m not sure any other CS program in the country has had that kind of jump like we have had over the past 35 years,” Stasko said. “The higher you go, the harder it is to jump even one spot.

“I think we knew that (the College) was going to grow and that was part of the plan. I’m not sure I would’ve envisioned we’d ever be 150 to 200 faculty in the college, but we could all see computer science was going to be a crucial part of society going forward.”

Stasko will continue to be a part of the School of IC as Professor Emeritus. His final student, Alexander Bendeck, finishes his Ph.D. in 2026. 

Bendeck will be the 25th student Stasko has advised and graduated over his career. He said he never had the funding to run a large lab, but that allowed him to invest in the students he took under his wing.

“I often found some unconventional Ph.D. students,” Stasko said. “Some of my early students started in very different areas of computer science. I’ve looked for diamonds in the rough. 

“I see some of them now with their families and they make me feel old because they have kids who are in college now. But they’ve done well. I think half of my students have gone into academia, and the other half into industry. I’m very proud in all that they’ve achieved, both personally and professionally.”

The Aarhus Conference, hosted by Aarhus University in Denmark, has been held every decade since 1975, addressing the most urgent and vital issues in computing worldwide. 

The latest conference, titled Computing (X) Crisis, took place in August and featured presentations, critiques, and workshops that explored computing’s influence on the human condition in a world filled with crises.

Assistant Professor Cindy Lin, Associate Professor Lynn Dombrowski, and School of Interactive Computing Professor and Chair Shaowen Bardzell authored the paper Whose, Which, and What Crisis? A Critical Analysis of Crisis in Computing Supply Chains. It was one of only 15 papers selected by conference organizers.

In the paper, in which Lin is credited as the lead author, the researchers advance a theoretical framework for understanding crises that impact the computing supply chain.

Bardzell, who served as program chair of the 2015 Aarhus Conference, approached Dombrowski and Lin about collaborating on a paper submission. Bardzell said the conference gets more than 100 submissions and has a minuscule acceptance rate.

“I knew I was going to go no matter what because I enjoyed it so much 10 years ago,” Bardzell said. “I was fortunate to come together with Lynn and Cindy. We spent six months reading, thinking, and debating together every week, and it was a pleasure to write it together.”

The authors identified common themes in areas they were already researching and examined how these themes affected the computing supply chain.

“We wanted to think about what this word means in relation to computing,” Dombrowski said. “Who gets to take advantage of a crisis, or who can construct a crisis in relation to computing? What’s not being talked about when we use that word?”

Lin is studying the rise of data centers and their impact on the environment and consumers. Dombrowski is an expert on the labor market and unjust labor practices. Bardzell has conducted extensive research on how chip manufacturing affects farming and agriculture in her homeland of Taiwan.

“We don’t often think about computing research as intergenerational colleagues working together,” Lin said. “I feel like the three of us represent very interesting generations of computing research that’s tied to critically thinking about the social and political aspects of computing. Each of us has different ways of thinking about those things.”

In the paper, the three authors discuss the concept of “against crisis thinking,” which emphasizes that crises affecting the computing supply chain aren’t self-evident phenomena. Human-computer interaction scholars, they say, should pay more attention to how the word “crisis” is introduced into public discourse and how it can be exploited by powerful actors and impact marginalized communities.

“Some players get to declare what is a crisis and whom it affects,” Lin said. “They create solutions to resolve the crisis, but they might not address what a chronic experience of a crisis may be.”

Although Bardzell said she considers it an honor to present at a conference that is so selective and is held only once a decade, she was encouraged to be among researchers dedicated to solving pressing societal and planetary issues.

“Academia can appear as a cutthroat environment where you’re trying to establish your brand and be known for XYZ,” Bardzell said. “At Aarhus, there was a strong sense of community and working alongside each other, and we’re better because of the people who work alongside us.”

Lin agreed and said that participating in Aarhus is different from the annual conferences where the researchers normally submit papers. 

“There’s something special about reflecting every 10 years,” Lin said. “It shows how much has changed but also how much things have remained the same.”

Danfei Xu, an assistant professor in Georgia Tech’s School of Interactive Computing, is introducing new models that provide robots with “on-the-job” training.

The National Science Foundation (NSF) awarded Xu its CAREER award given to early career faculty. The award will enable Xu to expand his research and refine his models, which could accelerate the process of robot deployment and alleviate manufacturers from the burden of achieving perfection.

“The main problem we’re trying to tackle is how to allow robots to learn on the job,” Xu said. “How should it self-improve based on the performance or the new requirements or new user preferences in each home or working environment? You cannot expect a robot manufacturer to program all of that.

“The challenging thing about robotics is that the robot must get feedback from the physical environment. It must try to solve a problem to understand the limits of its abilities so it can decide how to improve its own performance.”

As with humans, Xu views practice as the most effective way for a robot to improve a skill. His models train the robot to identify the point at which it failed in its task performance.

“It identifies that skill and sets up an environment where it can practice,” he said. “If it needs to improve opening a drawer, it will navigate itself to the drawer and practice opening it.”

The models allow the robot to split tasks into smaller parts and evaluate its own skill level using reward functions. Cooking dinner, for example, can be divided into steps like turning on the stove and opening the fridge, which are necessary to achieve the overall goal.

“Planning is a complex problem because you must predict what’s going to happen in the physical world,” Xu said. “We use machine learning techniques that our group has developed over the past two years, using generated models to generate positive futures. They’re very good at modeling long-horizon phenomena.

“The robot knows when it’s failed because there’s a value that tells it how well it performed the task and whether it received its reward. While we don’t know how to tell the robot why it failed, we have ways for it to improve its skills based on that measurement.” 

One of the biggest barriers that keeps many robots from being made available for public use is the pressure on manufacturers to make the robot as close to perfect as possible at deployment. Xu said it’s more practical to accept that robots will have learning gaps that need to be filled and to implement more efficient real-world learning models.

“We work under the pressure of getting everything correct before deployment,” he said. “We need to meet the basic safety requirements, but in terms of competence, it is difficult to get that perfect at deployment. This takes some of the pressure off because it will be able to self-adapt.”

Virtual Workspace for Data Workers

Yalong Yang, another assistant professor in the School of IC, also received the NSF CAREER Award for a research proposal that will design augmented and virtual reality (AR/VR) workspaces for data workers. 

“In 10 years, I envision everyone will use AR/VR in their office, and it will replace their laptop or their monitor,” Yang said.

Yang said he is also working with Google on the project and using Google Gemini to bring conventional applications to immersive space, with data tools being the most complicated systems to re-design for immersive environments.

The immersive workspace and interface will also enable teams of data workers to collaborate and share their data in real-time.

“I want to support the end-to-end process,” Yang said. “We have visualization tools for data, but it’s not enough. Data science is a pipeline — from collecting data to processing, visualizing, modeling and then communicating. If you only support one, people will need to switch to other platforms for the other steps.”

Yang also noted that prior research has shown that VR can enhance cognitive abilities, such as memory and attention and support multitasking. The results of his project could lead to maximizing worker efficiency without them feeling strained.

“We all have a cognitive limit in our working memory. Using AR/VR can increase those limits and process more information. We can expand people’s spatial ability to help them build a better mental model of the data presented to them.”

Yang was also recently named a 2025 Google Research Scholar as he seeks to build a new artificial intelligence (AI) tool that converts mobile apps into 3D immersive environments.

Workshops hosted by the Robert C. Williams Museum of Papermaking and led by Georgia Tech Assistant Professor HyunJoo Oh will introduce about 60 students from Atlanta Public Schools to paper-based electronics through hands-on workshops.

The Williams Museum will open an exhibit titled “The Future of Paper” that displays designs created in the workshop alongside visionary examples of paper-based technologies from Georgia Tech researchers.

The exhibit, funded by the National Science Foundation, is slated to open to the public in 2027.

Oh is a researcher with joint appointments in the School of Interactive Computing and the School of Industrial Design.She leads the Computational Design and Craft (CoDe Craft) Group at Georgia Tech, where her team integrates everyday craft materials with computing to support creative exploration.

Oh believes paper could be widely used to support prototyping printed circuit boards (PCBs) as a sustainable alternative to silicon. While silicon is the most prominent material used by technology companies to build computer chips, it isn’t biodegradable. And it can be harmful to the environment and contribute to e-waste. 

Paper, however, provides an eco-friendly platform for printing conductive traces and mounting small electronic components. With the expansion of printed electronic tools and techniques, paper and similar materials have become more popular among technologists who develop sensing technologies and wearable devices.

“It’s widely available and accessible,” Oh said. “I can’t think of anything more affordable and approachable that young makers and the broader maker community can use for circuits than paper.

“Printed electronics traditionally required expensive equipment, but with recent innovation in materials science, conductive materials such as conductive pens and paint available in local arts and crafts stores can be used to build circuits on paper. We can also print circuits using a regular office inkjet printer with silver ink.”

Shared Vision

Shortly after arriving at Georgia Tech in 2019, Oh knew she had to develop a project that would let her partner with the Williams Museum. 

“I was captivated by the museum’s space and its celebration of paper,” she said. “I wanted a collaboration that would integrate technology in a way that complemented and respected the museum’s existing beauty.”

Museum director Virginia Howell said the project was a perfect match for the museum, which has documented the history of papermaking since it was founded in 1939 by the Massachusetts Institute of Technology. Georgia Tech became the new home of the museum in 2003.

With more than 100,000 objects in its collection — some dating back as far as 2,000 years ago — the museum is unique, Howell said. Most papermaking museums are typically located at an historic mill, but the Williams Museum covers the history of papermaking.

Howell said that before she met Oh, she had been looking for an exhibit that would display the possible future of papermaking.

“We do the past of paper fantastically well, and we do the present of paper well through our changing exhibitions,” Howell said. “The future of paper is something we haven’t spent a lot of time interpreting.”

Crafting the Future

Oh and Howell agree that young people will shape that future. Oh said paper is commonly linked to art in the education sphere. As the material’s use in technology increases, however, it can funnel the interests of students toward engineering and computing. 

Incorporating paper and craft materials can invite more students to explore engineering and computing concepts. After all, a circuit board created on paper isn’t so different from one built on a silicon PCB, Oh said.

“This approach can excite the kind of students who usually feel disconnected from electronics and computing,” she said. “It gives those who only see themselves as creative or artistic a way to enjoy technology and resonate with it.

“Usually when I work with young students, especially girls, if I start with something technical, their interest wanes. But when I present those same ideas through art using familiar materials like paper, they become more engaged and confident. That’s when they start to flourish.”

Oh and Howell will hold three rounds of 10-week workshops for the students — spring 2026, fall 2026, and spring 2027. The best designs from those workshops will be displayed in the exhibit.

“They’ll feel more comfortable with computing and engineering as an introductory experience,” Howell said. “When they successfully build on it and realize they did this on a sheet of paper, it’s exciting to think what they’ll do when they get more sophisticated tools and access.”

With the help of a $1.25 million grant from the National Science Foundation, the three institutions seek to increase youth input into policymaking and encourage youth-led community organizing.

Youth-designed civic technologies are an effective way to engage youth with their communities, said Andrea Parker, an associate professor in Georgia Tech’s School of Interactive Computing. 

Examples of civic technologies are public data initiatives, citizen science projects, public issue reporting platforms, and digital voting platforms. Parker said the perspectives of young people are often neglected in the design of such technologies.

“We don’t know much about what community issues are important to youth because we haven’t asked them,” she said. “What is their vision for community well-being, and what do they want to address through civic technology?”

Parker is the lead principal investigator (PI) on the project that will engage youth from low socio-economic communities in Atlanta and Boston. She said the youth will decide what technologies will be created, but they could include a mobile app or a publicly accessible platform.

“We’re interested in studying how technologies can help youth become more civically engaged in their communities and build social connection, trust, and belonging amongst neighbors,” she said. 

“Youth in lower-income neighborhoods face increased threats to their mental health. Socially cohesive communities can counteract those barriers and are essential for youth well-being.”

Parker added that impoverished communities often have less social cohesion compare to wealthier areas. Higher-income neighborhoods often have more access to resources that support social cohesion and civic engagement. 

Backed by Data

Brooke Foucault Welles, co-PI, professor, and interim dean at Northeastern’s College of Media, Arts and Design, said she’s interested in seeing which issues the youths from both Atlanta and Boston will address through their design process. Studying and working with youth across these geographic settings will help the team identify how civic technology can best support youth in varied neighborhood contexts.

The project will also advance data literacy among young people as they collect and study data to support the new technologies. Welles said data-centered advocacy increases young people’s chances of being heard by elder community members.

“Empowering young people to use data when they’re making their arguments about what matters to them and to their communities is the point of this project,” she said. “It makes their arguments more compelling if they can present data to the adult members of their communities about what’s going on.”

The project’s reach could expand beyond Atlanta and Boston.

Once the technologies are designed, the researchers will package them and make them publicly available as a toolkit. 

If successful, the project could drive a movement toward more collective organizing to ensure the youth perspective gets factored into community decision-making. 

“They’re a vital part of our communities, and they’re the ones for whom our decisions have the biggest impact,” Welles said. “These are the times when they’re forming their own civic identities, so engaging them in civic life has long ripple effects. We create more active and thoughtful citizens when we engage young people with civic life.”

A new approach proposed by Tech researcher Yalong Yang uses generative artificial intelligence (GenAI) technologies to convert almost any mobile or web-based app into a 3D environment. 

That includes application software programs from Microsoft and Adobe as well as any social media (Tiktok), entertainment (Spotify), banking (PayPal), or food service app (Uber Eats) and everything in between.

Yang aims to make the 3D environments compatible with augmented and virtual reality (AR/VR) headsets and smart glasses. He believes his research could be a breakthrough in spatial computing and change how humans interact with their favorite apps and computer systems in general.

“We’ll be able to turn around and see things we want, and we can grab them and put them together,” said Yang, an assistant professor in the School of Interactive Computing. “We’ll no longer use a mouse to scroll or the keyboard to type, but we can do more things like physical navigation.”

Yang’s proposal recently earned him recognition as a 2025 Google Research Scholar. Along with converting popular social apps, his platform will be able to instantly render Photoshop, MS Office, and other workplace applications in 3D for AR/VR devices.

“We have so many applications installed in our machines to complete all the various types of work we do,” he said. “We use Photoshop for photo editing, Premiere Pro for video editing, Word for writing documents. We want to create an AR/VR ecosystem that has all these things available in one interface with all apps working cohesively to support multitasking.”

Filling The Gap With AI

Just as Google’s Veo and Open AI’s Sora use generative-AI to create video clips, Yang believes it can be used to create interactive, immersive environments for any Android or Apple app. 

“A critical gap in AR/VR is that we do not have all those existing applications, and redesigning all those apps will take forever,” he said. “It’s urgent that we have a complete ecosystem in VR to enable us to do the work we need to do. Instead of recreating everything from scratch, we need a way to convert these applications into immersive formats.”

AfriMed-QA, developed by researchers from Georgia Tech and Google, could reduce the burden on African healthcare systems. 

The researchers said people in need of medical care file into overcrowded clinics and hospitals and face excruciatingly long waits with no guarantee of admission or quality treatment. There aren’t enough trained healthcare professionals available to meet the demand.

Some healthcare question-answer chatbots have been introduced to treat those in need. However, the researchers said there’s no transparent or standardized way to test or verify their effectiveness and safety.

The dataset will enable technologists and researchers to develop more robust and accessible healthcare chatbots tailored to the unique experiences and challenges of Africa. 

One such new tool is Google’s MedGemma, a large-language model (LLM) designed to process medical text and images. AfriMed-QA was used for training and evaluation purposes.

AfriMed-QA stands as the most extensive dataset that evaluates LLM capabilities across various facets of African healthcare. It contains 15,000 question-answer pairs culled from over 60 medical schools across 16 countries and covering numerous medical specialties, disease conditions, and geographical challenges. 

Tobi Olatunji and Charles Nimo co-developed AfriMed-QA and co-authored a paper about the dataset that will be presented at the Association for Computational Linguistics (ACL) conference next week in Vienna.

Olatunji is a graduate of Georgia Tech’s Online Master of Science in Computer Science (OMSCS) program and holds a Doctor of Medicine from the College of Medicine at the University of Ibadan in Nigeria. Nimo is a Ph.D. student in Tech’s School of Interactive Computing, where he is advised by School of IC professors Michael Best and Irfan Essa.

Focus on Africa

Nimo, Olatunji, and their collaborators created AfriMed-QA as a response to MedQA, a large-scale question-answer dataset that tests the medical proficiency of all major LLMs. That includes Google’s Gemini, OpenAI’s ChatGPT, and Anthropic’s Claude, among others.

However, because MedQA is trained solely on the U.S. Medical License Exams, Nimo said it is not adequate to serve patients in underdeveloped African countries nor the Global South at-large.

“AfriMed-QA has the contextualized and localized understanding of African medical institutions that you don’t get from Med-QA,” Nimo said. “There are specific diseases and local challenges in our dataset that you wouldn't find in any U.S.-based dataset.”

Olatunji said one problem African users may encounter using LLMs trained on MedQA is that they may advise unfeasible treatments or unaffordable prescription drugs.

“You consider the types of drugs, diagnostics, procedures, or therapies that exist in the U.S. that are quite advanced. These treatments are much more accessible, for example in the US, and Europe,” Olatunji said. “But in Africa, they’re too expensive and many times unavailable. They may cost over $100,000, and many people have no health insurance. Why recommend such treatments to someone who can’t obtain them?”

Another problem may be that the LLM doesn’t take a medical condition seriously if it isn’t predominant in the U.S.

“We tested many of these models, for example, on how they would manage sickle-cell disease signs and symptoms, and they focused on other “more likely” causes and did not rank or consider sickle cell high enough as a possible cause,” he said. “They, for example, don’t consider sickle-cell as important as anemia and cancer because sickle-cell is less prevalent in the U.S.”

In addition to sickle-cell disease, Olatunji said some of the healthcare issues facing Africa that can be improved through AfriMed-QA include:

• HIV treatment and prevention
• Poor maternal healthcare
• Widespread malaria cases
• Physician shortage
• Clinician productivity and operational efficiency

Google Partnership

Mercy Asiedu, senior author of the AfriMed-QA paper and research scientist at Google Research, has dedicated her career to improving healthcare in Africa. Her work began as a Ph.D. student at Duke University, where she invented the Callascope, a groundbreaking non-invasive tool for gynecological examinations

With her current focus on democratizing healthcare through artificial intelligence (AI), Asiedu, who is from Ghana, helped create a research consortium to develop the dataset. The consortium consists of Georgia Tech, Google, Intron, Bio-RAMP Research Labs, the University of Cape Coast, the Federation of African Medical Students Association, and Sisonkebiotik.

Sisonkebiotik is an organization of researchers that drives healthcare initiatives to advance data science, machine learning, and AI in Africa.

Olatunji leads the Bio-RAMP Research Lab, a community of healthcare and AI researchers, and he is the founder and CEO of Intron, which develops natural-language processing technologies for African communities.

In May, Google released MedGemma, which uses both the MedQA and Afri-MedQA datasets to form a more globally accessible healthcare chatbot. MedGemma has several versions, including 4-billion and 27-billion parameter models, which support multimodal inputs that combine images and text.

“We are proud the latest medical-focused LLM from Google, MedGemma, leverages AfriMed-QA and improves performance in African contexts,” Asiedu said. 

“We started by asking how we could reduce the burden on Africa’s healthcare systems. If we can get these large-language models to be as good as experts and make them more localized with geo-contextualization, then there’s the potential to task-shift to that.”

The project is supported by the Gates Foundation and PATH, a nonprofit that improves healthcare in developing countries.

Georgia Tech roboticists are attempting to identify the most stressful periods that human teleoperators experience while performing tasks remotely. A novel study provides new insights into determining when a teleoperator needs to operate at a high level of focus and which parts of the task can be delegated to robot automation.

School of Interactive Computing Associate Professor Matthew Gombolay calls it the “sweet spot” of human ingenuity and robotic precision. Gombolay and students from his CORE Robotics Labconducted a novel study that measures stress and workload on human teleoperators.

Gombolay said it can inform military officials on how to strategically implement task automation and maximize human teleoperator performance.

Humans continue to hand over more tasks to robots to perform, but Gombolay said that some functions will still require human input and oversight for the foreseeable future.

Specific applications, such as space exploration, commercial and military aviation, disaster relief, and search and rescue, pose substantial safety concerns. Astronauts stationed on the International Space Station, for example, manually control robots that bring in supplies, move cargo, and make structural repairs.

“It’s brutal from a psychological perspective,” Gombolay said.

The question often asked about automating a task in these fields is, at what point can a robot be trusted more than a human?

A recent paper by Gombolay and his current and former students — Sam Yi Ting, Erin Hedlund-Botti, and Manisha Natarajan — sheds new light on the debate. The paper was published in the IEEE Robotics and Automation Letters and will be presented at the International Conference on Robotics and Automation in Atlanta.

The NASA-funded study can identify which aspects of tedious, time-consuming tasks can be automated and which require human supervision. If roboticists can pinpoint the elements of a task that cause the least stress, they can automate these components and enable humans to oversee the more challenging aspects.

“If we’re talking about repetitive tasks, robots do better with that, so if you can automate it, you should,” said Ting, a former grad student and lead author of the paper. “I don’t think humans enjoy doing repetitive tasks. We can move toward a better future with automation.”

Military officials, for example, could measure the stress of remote drone pilots and know which times during a pilot’s shift require the highest level of attention.

“We can get a sense of how stressed you are and create models of how divided your attention is and the performance rate of the tasks you’re doing,” Gombolay said.

“It can be a low-stress or high-stress situation depending on the stakes and what’s going on with you personally. Are you well-caffeinated? Well-rested? Is there stress from home you’re bringing with you to the workplace? The goal is to predict how good your task performance will be. If it indicates it might be poor, we may need to outsource work to other people or create a safe space for the operator to destress.”

The Stress Test

For their study, the researchers cut a small river-shaped path into a medium-density fiberboard. The exercise required the 24 participants to use a remote robotic arm to navigate through the path from one end to the other without touching the edges.

The experiment grew more challenging as new stress conditions and workload requirements were introduced. The changing conditions required the test participants to multitask to complete the assignment.

Gombolay said the study supports the Yerkes-Dodson Law, which states that moderate levels of stress increase human performance.

The experiment showed that operators felt overwhelmed and performed poorly when multitasking was introduced. Too much stress led to poor performance, but a moderate amount of stress induced more engagement and enhanced teleoperator focus. 

Ting said finding that ideal stress zone can lead to a higher performance rating. 

“You would think the more stressed you are, the more your performance decreases,” Ting said. “Most people didn’t react that way. As stress increased, performance increased, but when you increased workload and gave them more to do, that’s when you started seeing deteriorating performance.”

Gombolay said no stress can be just as detrimental as too much stress. Performing a task without stress tends to cause teleoperators to become disinterested, especially if it is repetitive and time-consuming.

“No stress led to complacency,” Gombolay said. “They weren’t as engaged in completing the task.

“If your excitement is too low, you get so bored you can’t muster the cognitive energy to reason about robot operation problems.”

The Human Factor

Roboticists have made significant leaps in recent years to remove teleoperators from the equation. Still, Gombolay said it’s too early to tell whether robots can be trusted with any task that a human can perform.

“We’re a long way from full autonomy,” he said. “There’s a lot that robots still can’t do without a human operator. Search and rescue operations, if a building collapses, we don’t have much training data for robots to go through rubble by themselves to rescue people. There are ethical needs for humans to be able to supervise or take direct control of robots.”

As Marines, Moreno and Santacruz cultivated expertise in precision and mission-critical support for on-the-ground forces. That experience helps them streamline the administrative process of the School of Interactive Computing as they secure research grants that improve people’s lives.

The two work as faculty support coordinators in the School of IC. They first met in the Marines in 2019 while assigned to the Personnel Retrieval and Processing Company of the 4th Marine Logistics Group in Smyrna, Ga. 

Moreno is still in the Navy reserves and holds the rank of petty officer. Santacruz held the rank of sergeant and was the administration chief when Moreno joined the company. He was discharged in 2022.

The Personnel Retrieval and Processing Company is responsible for the recovery, processing, and preparation of the bodies of fallen service members. The unit, which has detachments domestically and overseas, handles the mortuary affairs, documentation, transportation, and the processing of remains and personal effects.

Moreno and Santacruz were responsible for coordinating travel and deployments, as well as processing legal and medical documents. 

Before Smyrna, they gained administrative experience working in foreign nations and conflict zones.

Moreno joined the Marines out of high school in 2010. After a stint at Marine Corps Air Station in Jacksonville, N.C., he was assigned to the administrative staff of the U.S. Embassy in Abu Dhabi, United Arab Emirates. He then transferred to Camp Pendleton in California before being assigned to Combat Logistics Regiment 27, 2nd Marine Logistics Group.

Santacruz enlisted in 2014 and was also assigned to the Combat and Logistics Regiment 27. In 2016, he deployed on a six-month tour in Djibouti, where he supported combat operations and civilian evacuation efforts in nearby conflict zones.

In 2021, Moreno decided to join the reserves and pursue a professional career in administration. He immediately received a call back after submitting his application to Georgia Tech.

Since they still lived in Atlanta, Moreno and Santacruz kept in touch with each other. When Moreno heard Santacruz had left the Marines, he called him and encouraged him to apply to Georgia Tech.

“We still keep up with other friends who were stationed with us,” Moreno said. “The brotherhood doesn’t end in the service.”

As faculty support coordinators, they process all the necessary paperwork for grant applications to government organizations that fund research, such as the National Science Foundation (NSF). They also coordinate travel for faculty and students to various conferences and events.

Moreno and Santacruz said they enjoyed working behind the scenes in the Marines knowing everything they did was critical to the success of the units they supported.

They brought that mission-first mindset with them to Georgia Tech.

“The most rewarding thing is being able to see the fruits of our work,” Santacruz said. “When Dean (Vivek) Sikar sends emails congratulating students and faculty, we see those names, and we’re the ones who got that spend authorization for them. You see the stuff you’re working for and you know it’s changing something either at Tech or even globally.”

Moreno said Georgia Tech encourages work-life balance, and the School of Interactive Computing staff supports him when he’s required to fulfill his duties in the reserves. He left the School for seven months on active-duty orders over 2023 and 2024 at the Navy Reserve Center in Marietta.

He said he never had to worry about his job at Tech while he was gone. 

“I love that Georgia Tech gives me the ability to balance both,” he said.

He also said he enjoys taking on challenges that arise during the day.

“We always joke that every day is different here in Interactive Computing,” Moreno said. “There’s always a different challenge, a different scenario.

“There’s more flexibility here. There are a lot of ways to get something done. You can have conversations about different ideas.”

Top South Korean research institutes have enlisted Georgia Tech researchers Sehoon Ha and Jennifer G. Kim to develop artificial intelligence (AI) to help the humanoid assistant navigate hospitals and interact with doctors, nurses, and patients.

Ha and Kim will partner with Neuromeka, a South Korean robotics company, on a five-year, 10 billion won (about $7.2 million US) grant from the South Korean government. Georgia Tech will receive about $1.8 million of the grant.

Ha and Kim, assistant professors in the School of Interactive Computing, will lead Tech’s efforts and also work with researchers from the Korea Advanced Institute of Science and Technology and the Electronics and Telecommunications Research Institute.

Neuromeka has built industrial robots since its founding in 2013 and recently decided to expand into humanoid service robots.

Lee, the group leader of the humanoid medical assistant project, said he fielded partnership requests from many academic researchers. Ha and Kim stood out as an ideal match because of their robotics, AI, and human-computer interaction expertise. 

For Ha, the project is an opportunity to test navigation and control algorithms he’s developed through research that earned him the National Science Foundation CAREER Award. Ha combines computer simulation and real-world training data to make robots more deployable in high-stress, chaotic environments. 

“Dr. Ha has everything we want to put into our system, including his navigation policies,” Lee said. “He works with robots and AI, and there weren’t many candidates in that space. We needed a collaborator who can create the software and has experience running it on robots.”

Ha said he is already considering how his algorithms could scale beyond hospitals and become a universal means of robot navigation in unstructured real-world environments.

“For now, we’re focusing on a customized navigation model for Korean environments, but there are ways to transfer the data set to different environments, such as the U.S. or European healthcare systems,” Ha said. 

“The final product can be deployed to other systems and industries. It can help industrial workers at factories, retail stores, any place where workers can get overwhelmed by a high volume of tasks.”

Kim will focus on making the robot’s design and interaction features more human. She’ll develop a large-language model (LLM) AI system to communicate with patients, nurses, and doctors. She’ll also develop an app that will allow users to input their commands and queries. 

“This project is not just about controlling robots, which is why Dr. Kim’s expertise in human-computer interaction design through natural language was essential.,” Lee said. 

Kim is interviewing stakeholders from three South Korean hospitals to identify service and care pain points. The issues she’s identified so far relate to doctor-patient communication, a lack of emotional support for patients, and an excessive number of small tasks that consume nurses’ time.

“Our goal is to develop this robot in a very human-centered way,” she said. “One way is to give patients a way to communicate about the quality of their care and how the robot can support their emotional well-being.

“We found that patients often hesitate to ask busy nurses for small things like getting a cup of water. We believe this is an area a robot can support.”

The robot’s hardware will be built in Korea, while Ha and Kim will develop the software in the U.S.

Jong-hoon Park, CEO of Neuromeka, said in a press release the goal is to have a commercialized product as soon as possible. 

“Through this project, we will solve problems that existing collaborative robots could not,” Park said. “We expect the medical AI humanoid robot technology being developed will contribute to reducing the daily work burden of medical and healthcare workers in the field.”

Meta announced in February its partnership with the labs of professors Danfei Xu and Judy Hoffman on a novel computer vision-based algorithm called EgoMimic. It enables robots to learn new skills by imitating human tasks from first-person video footage captured by Meta’s Aria smart glasses. 

Xu’s Robot Learning and Reasoning Lab (RL2) displayed EgoMimic in action at ICRA May 19-23 at the World Congress Center in Atlanta.

Lawrence Zhu, Pranav Kuppili, and Patcharapong “Elmo” Aphiwetsa — students from Xu’s lab — used Egomimic to compete in a robot teleoperation contest at ICRA. The team finished second in the event titled What Bimanual Teleoperation and Learning from Demonstration Can Do Today, earning a $10,000 cash prize.

Teams were challenged to perform tasks by remotely controlling a robot gripper. The robot had to fold a tablecloth, open a vacuum-sealed container, place an object into the container, and then reseal it in succession without any errors.

Teams completed the tasks as many times as possible in 30 minutes, earning points for each successful attempt.

The competition also offered different challenge levels that increased the points awarded. Teams could directly operate the robot with a full workstation view and receive one point for each task completion. Or, as the RL2 team chose, teams could opt for the second challenge level.

The second level required an operator to control the task with no view of the workstation except for what was provided to through a video feed. The RL2 team completed the task seven times and received double points for the challenge level.

The third challenge level required teams to operate remotely from another location. At this level, teams could earn four times the number of points for each successful task completed. The fourth level challenged teams to deploy an algorithm for task performance and awarded eight points for each completion.

Using two of Meta’s Quest wireless controllers, Zhu controlled the robot under the direction of Aphiwetsa, while Kuppili monitored the coding from his laptop.

“It’s physically difficult to teleoperate for half an hour,” Zhu said. “My hands were shaking from holding the controllers in the air for that long.”

Being in constant communication with Aphiwetsa helped him stay focused throughout the contest.

“I helped him strategize the teleoperation and noticed he could skip some of the steps in the folding,” Aphiwetsa said. “There were many ways to do it, so I just told him what he could fix and how to do it faster.”

Zhu said he and his team had intended to tackle the fourth challenge level with the EgoMimic algorithm. However, due to unexpected time constraints, they decided to switch to the second level the day before the competition due to unexpected time constraints. 

“I think we realized the day before the competition training the robot on our model would take a huge amount of time,” Zhu said. “We decided to go for the teleoperation and started practicing.”

He said the team wants to tackle the highest challenge level and use a training model for next year’s ICRA competition in Vienna, Austria.

ICRA is the world’s largest robotics conference, and Atlanta hosted the event for the third time in its history, drawing a record-breaking attendance of over 7,000.

She said the police officer who interviewed her was dismissive and neglected to write down details that Foriest found significant. The deficient police report was picked up by local media, which led to news stories that inaccurately described the crime and left out important information.

Foriest said she learned from the incident that incomplete information doesn’t mitigate violence. The perspectives and stories of people who experience violence are essential to reliable data.

The incident guided Foriest as she committed to research that gathers complete and accurate data on multiple types of violence, including violent injury and homicide, intimate partner violence, gender-based violence, and suicide.

Foriest earned a bachelor’s in health science from Columbus State University. She also holds two master’s degrees: one in public health from the University of Southern California, and another in technology leadership and management from Agnes Scott College.

In 2021, Foriest started her Ph.D. in human-centered computing at Georgia Tech to understand how technology influences violence.

“I look at all types of violence as an outcome of how technology affects communication,” she said. 

One thing she discovered was that even though technology can amplify victims’ voices, it is often used to silence them.

“The same social dynamics that keep people from disclosing their violent experiences to formal reporting sources offline also happen online,” she said.

Bringing the Cardiff Model to the U.S.

Before arriving at Tech, Foriest worked for eight years as an injury prevention coordinator at Grady Memorial Hospital in Atlanta. She implemented a trauma recovery center and Atlanta’s first hospital-based violence intervention program.

While in that position, she worked with the Cardiff Model for Violence Prevention, a public health approach to violence prevention developed by researchers at Cardiff University in Wales.

The Cardiff model’s philosophy is that violence prevention is best achieved when the healthcare and law enforcement sectors combine geographical data to determine where violence occurs in a community. 

“The Cardiff model taught Wales there was a lot about violence they didn’t know from police data alone,” Foriest said. 

One example is that researchers learned an alarming number of hospital patients were brought in from local taverns. This finding informed policymakers to implement new regulations, such as changing licensing requirements and serving alcohol in toughened glasses or non-glass vessels so they can’t be used as weapons.

In 2011, the city of Cardiff reported a 42% reduction in hospital admissions for hospital injuries. It wasn’t long before the researchers in the U.S. began importing the Cardiff model. In 2018, it became an official policy of the Centers for Disease Control and Prevention (CDC). 

The U.S. Department of Justice found in 2022 that 58% of violent crimes were not reported to law enforcement. Sixteen cities that make up the Cardiff Model for Violence Prevention National Networkare now gathering and mapping patient-reported violent injury data from hospitals to fill that data gap. 

Atlanta is one of the cities in that network, and Foriest has been an on-the-ground researcher collecting that data. Her work with the Cardiff model seamlessly integrated into her Ph.D. research as she sought ways to turn technology into a safe avenue of violence disclosure.

Working with Alex Godwin, a former Ph.D. student at Georgia Tech who is now an assistant professor at American University, she helped develop a user interface and mapping algorithm. The tool allows hospital patients who are violence victims to identify the location of the violent incident they experienced.

Foriest said, “Around the Covid-19 pandemic, we had challenges getting patients screened, and we thought we should explore different options.

“Our interface allows patients to tap down to the degree they’re comfortable on the geographic location where they were injured.

“It improved our ability to map data tremendously and decreased some of the risks patients face when disclosing violence.”

Foriest and Godwin's paper on the development of the interface tool earned an honorable mention for best paper at the 2025 Conference on Human Factors in Computing Systems (CHI) in Yokohama, Japan.

Foriest also co-authored an award-winning paper at the 2024 Conference on Computer-Supported Cooperative Work (CSCW). That paper examined how social media often silences violence victims.

Foriest is also a fellow for Data Science and Innovation at the CDC, where she continues her work on the Cardiff model. She also examines how news media coverage of suicides can often reinforce stigmas about the causes of suicide in that role.

Thriving at Tech

Foriest is entering her fifth year as a Ph.D. student, but before she came to Tech, she had no computing experience. She applied to numerous Ph.D. programs but was eventually persuaded that technology could complement her public health expertise and her goal of preventing violence.

“Tech was the only place where I could gain a new skill set while doing the things that I wanted to do in research,” she said. 

“That felt like the best fit for me, where I would get the most out of my training. I was encouraged by faculty and my peers to recognize that my perspective is valuable, and I can speak from that place and bridge my knowledge with HCI concepts.”

It may not be long before augmented reality/virtual reality (AR/VR) headsets cause them to keep their phones in their pockets when they want to read The New York Times or The Washington Post.

Data visualization and AR/VR researchers at Georgia Tech are exploring how users can interact with news stories through AR/VR headsets and are determining which stories are best suited for virtual presentation.

Tao Lu, a Ph.D. student at the School of Interactive Computing, Assistant Professor Yalong Yang, and Associate Professor Alex Endert led a recent study that they say is among the first to explore user preference in virtually designed news stories.

The researchers will present a paper they authored based on the study at the 2025 Conference on Human Factors in Computing Systems this week in Yokohama, Japan.

Digital platforms have elevated explanatory journalism, which provides greater context for a subject through data, images, and in-depth analysis. These platforms also allow stories to be more visually appealing through graphic design and animation. 

Lu said AR/VR can further elevate explanatory journalism through 3D, interactive spatial environments. He added that media organizations should think about how the stories they produce will appear in AR/VR as much as they think about how they will appear on mobile devices.

“We’re giving users another option to experience the story and for designers and developers to show their stories in another modality,” Lu said.

“A screen-based story on a smartphone is easy to use and cost-effective. However, some stories are better presented in AR/VR, which will become more popular as technology gets cheaper. AR/VR can provide 3D spatial information that would be hard to understand on a phone or desktop screen.”

Active or Passive Interactions?

Using Meta’s Oculus Quest 3, the researchers and their collaborators created four immersive virtual reality simulations from web-based news stories produced by The New York Times:

• Why opening windows was key to classroom ventilation during the Covid-19 pandemic
• The destruction of Black homes and businesses in the Tulsa Race Massacre
• How climate change could create dramatic dangers in the Atlantic Ocean
• How 9/11 changed Manhattan’s financial district

The study aimed to determine whether users prefer to be actively or passively immersed in a story, whether from a first-person or third-person point of view, or a combination of these perspectives.

“We’re in the nascent stages of storytelling in VR,” said Endert, whose research specializes in data visualization. “We lack the design knowledge of which mode of immersion we should use if we want a certain reaction from the audience. Understanding design is at the crux of our study.”

Active immersion gives the user complete control over their experience. The classroom simulation offers a first-person point of view and allows users to teleport from one point in the classroom to another. New information from the story is presented each time the user moves to a new point. 

The researchers acknowledged they could design a free-roaming simulation that allows users to walk freely within the classroom. However, they restricted that ability for this study due to safety concerns and lab space constraints. 

In the Tulsa Race Massacre simulation, which uses a passive, first-person point of view, users follow a predefined route along one of Tulsa’s main streets. Information about each building is presented at each step.

The Atlantic Ocean simulation is an active, third-person experience. The user sees a representation of Earth and can select which interaction points to explore to learn new information.

The 9/11 simulation is a passive third-person experience. Each step includes a narrative paragraph with companion visual elements, and users proceed to the next step through a navigation trigger.

Finding the Right Balance

Lu said that first-person active enhances spatial awareness, while third-person passive improves contextual understanding. Journalists and VR designers must determine which presentation is most effective case by case.

Yang said the goal should be to balance interests in making those determinations, which might require compromise. Knowing how users prefer to consume news is critical, but journalists still have an editorial responsibility to decide what the public should know and how to present information.

“You have more freedom to explore in an active experience versus a passive experience,” Yang said. “But if you give them too much freedom, they might stray from your planned narrative and miss important information you think they should know. We want to understand how we can balance both ends of this spectrum and what the right level is that we can give people in storytelling.”

The study and others indicate that users retain information better when they feel like they are part of the story. Yang said the technology to make that possible isn’t there yet, but it’s coming along as wearable VR devices become more accessible.

The debate is whether these devices will become people's preferred technology for consuming content. According to the Pew Research Center, 86% of U.S. adults say they at least sometimes get their news from a smartphone, computer, or tablet.

“I believe AR and VR will be mainstream in the future and will replace everything, but I think there’s a transition period,” Yang said. “Older devices will exist and act as support. It’s an ecosystem.”

The crowd grew so large that no one could move in the alley, resulting in the deadliest crowd crush in the nation’s history. Nearly 160 people were killed, and another 196 were injured.

Soonho Kwon, a first-year human-centered computing Ph.D. student at Georgia Tech, lived within walking distance of the alley when the incident occurred.

“It was tragic,” Kwon said. “It really makes you think about how life is fragile. Everyone in my community talked about what it would have been like if they were in that alleyway.”

Many of the victims were young people — some of them teens who had no identification on them. Kwon thought about their family members being told their loved ones’ lives had been cut short. He wondered what memories those families would have of the deceased.

The incident inspired Kwon to create a new mobile platform that helps young adults and career professionals create a post-death memorial for their families. The platform, which Kwon and his research collaborators named Timeless, allows users to be remembered how they want to be remembered in the event of their untimely death.

“Most death preparation services are for terminally ill patients or aging adults, focusing on will management or funeral planning,” Kwon said. “We thought such needs may differ for young adults and asked how we could design a system that better caters to their needs.”

Timeless is a photo-based death preparation system that enables users to send a physical package containing pre-curated pictures, voice recordings, and letters to a designated recipient in the event of their passing. 

The system syncs with a user’s mobile photo album and creates a list of scanned faces. Users can select a face and view all the photos they’ve taken with that person. They can choose which photos they want sent to that person after death and write individual messages for each image.

Once the user’s death has been reported, Timeless sends a package to each selected individual with printed photos, letters, and a QR code or a CD that contains videos or voice recordings.

Breaking the Ice

Kwon and his collaborators designed Timeless based on a group study that asked participants to imagine what would happen if they unexpectedly died. The participants were asked what was on their bucket lists, their epitaphs, and what they would wish for if they could make one wish come true.

“Surprisingly, people were happy to participate because we framed it in a way that wasn’t gloomy,” Kwon said. “Many shared that reflecting on their death motivated them to actively express their love and be grateful for what they have. Turning something as heavy as death into something positive was a key design implication.”

Digital vs. Physical

Kwon began his research career examining virtual commemoration systems, including Facebook and Instagram commemoration pages, during the Covid-19 pandemic and exploring how technology can meaningfully memorialize the deceased.

He said two aspects distinguish Timeless from other commemoration platforms: 

• The deceased can decide how and by whom they want to be remembered.
• The fusion of digital memorialization with physical memorialization

“Leveraging only the digital side of it can be superficial,” Kwon said. “We build monuments, statues, and tombstones because the notion of death itself is losing your physical presence. By making it physical, we aimed to connect the discussion on digital legacies to traditional human commemoration forms.”

AI Afterlife

Kwon also said he is aware of artificial intelligence (AI) afterlife. This emerging technology allows people to train an AI agent and produce digital avatars with which family and friends can communicate after they die.

Meredith Ringel Morris, director and principal scientist for human-AI interaction at Google DeepMind, spoke about AI afterlife in October during the Summit on AI, Responsible Computing, and Society hosted by Georgia Tech’s School of Interactive Computing.

In her talk, Morris spoke about the criticism AI afterlife is already facing for causing people to experience extended grief and the inability to move on from losing a loved one.

Kwon said another drawback is that AI agents are susceptible to hallucinations and could say untrue things about the deceased. 

“How can you say for sure that the representation of AI is me?” he said. “As researchers, our role is to explore and critically examine how the emergence of such technology may shape society while striving to ensure its development benefits people.” 

Kwon sees Timeless as a catalyst for meaningful discussions about how a digital legacy curation system may accurately reflect a user’s wishes before death. 

He will present a paper on Timeless's design process and its implications at the 2025 ACM Conference on Human Factors in Computing Systems (CHI) this week in Yokohama, Japan.

Device prototypes like these usually require massive amounts of time and external resources to build, but thanks to the Uncommon Sense Lab, they can now be conveniently developed on Georgia Tech’s campus.

The lab is housed in Georgia Tech’s School of Interactive Computing and is managed by Assistant Professor Alexander Adams.

“Our overall goal is to give better access to healthcare,” Adams said. “We’re always looking at who we’re doing this for, how we’re getting it to them, how it addresses specific needs, and how to make it as financially accessible as possible.

“There’s always a space for high-end, high-precision equipment, but not everyone has access, and people are often afraid to get checked out because of the cost. If we can build something that doesn’t necessarily give someone a perfect measurement of a condition, but it can tell them they should go to the doctor, that might be enough to save a life.”

The lab provides resources to interdisciplinary researchers with backgrounds in computing, robotics, mechanical engineering, electrical engineering, and biomedical engineering to develop novel sensing and feedback system prototypes.

“We render physical prototypes that would be difficult to build without a centralized location for these resources,” said Adams, who is affiliated with the Institute for Robotics and Intelligent Machines and the Parker H. Petit Institute for Bioengineering and Bioscience. “We give students access to the tools and knowledge to build things that would typically seem unreachable.

“There’s nowhere else on campus with this collective that can go end-to-end from mechanical engineering to biomedical engineering to electrical engineering to usability.”

Examples of current prototypes being developed in the lab include a device that trains people with post-traumatic stress disorder to breathe in more regular patterns, and another that measures a person’s heart rate when they vape.

“We want to learn more about that behavior through these sensing devices, and then we’ll look at figuring out how we can help people correct their breathing patterns or quit their addiction,” Adams said.

The Uncommon Sense Lab offers high-tech, state-of-the-art machinery, including:

• 3D printers, including fused deposition modeling (FDM) printers for multi-material, high-precision prints
• A laser cutter for producing printed circuit boards (PCBs)
• Surface mount PCB manufacturing station with soldering tools, paste dispensers, and rework stations
• Optical work benches for optical system design, including microscopes and fluidics workstations
• Resin materials for casting and molding prosthetics
• Vacuum chambers and pressure chambers
• Saws, mills, lathes, and other mechanical tools for processing wood and soft metals
• Saws, grinders, polishers, and other wet tools for glass, stone, and ceramics

Since he started at the School of Interactive Computing in 2022, Adams has envisioned the lab. The lab space in the Technology Square Research Building in Midtown was thoroughly renovated, including access control, a new ceiling grid, environmental controls, pressurized air, plumbing, and vacuum and air filtration systems.

“This is the result of having built two labs at previous institutions, what I’ve learned about my type of work and my field, and what the most useful things are to handle our diverse projects,” he said.

“One of the reasons I came to Georgia Tech was because they saw the value of being interdisciplinary in a computing world and having a full lab space instead of just an office.” 

Adams said the lab will accelerate the timelines of current projects for the researchers who use it and create more bandwidth for them to take on more projects.

“I want my students to have everything at hand instead of waiting every time we need to do something,” he said. “This space is for someone who might have an idea for a remote diagnostic tool, but they’re wondering how to build it, add computation, and test it. This is the solution for those wondering how they can do that without spending a year finding and organizing access to facilities or ordering various parts.”

Adams said the lab is not a public space, but anyone interested in using it can make a written request for access. The work must be part of a collaboration, and faculty must provide funds to use resources. Access is contingent upon passing several safety courses and in-person training.

Current training methods limit robots from being produced at the necessary scale to put a robot in every home, said Simar Kareer, a Ph.D. student in the School of Interactive Computing.

“Traditionally, collecting data for robotics means creating demonstration data,” Kareer said. “You operate the robot’s joints with a controller to move it and achieve the task you want, and you do this hundreds of times while recording sensor data, then train your models. This is slow and difficult. The only way to break that cycle is to detach the data collection from the robot itself.”

[VIDEO: Meta Shares EgoMimic Case Study Video]

Other fields, such as computer vision and natural language processing (NLP), already leverage training data passively culled from the internet to create powerful generative AI and large-language models (LLMs).

Many roboticists, however, have shifted toward interventions that allow individual users to teach their robots how to perform tasks. Kareer believes a similar source of passive data can be established to enable practical generalized training that scales the production of humanoid robots.

This is why Kareer collaborated with School of IC Assistant Professor Danfei Xu and his Robot Learning and Reasoning Lab to develop EgoMimic, an algorithmic framework that leverages data from egocentric videos.

Meta’s Ego4D dataset inspired Kareer’s project. The benchmark dataset, released in 2023, consists of first-person videos of humans performing daily activities. This open-source data set trains AI models from a first-person human perspective.

“When I looked at Ego4D, I saw a dataset that’s the same as all the large robot datasets we’re trying to collect, except it’s with humans,” Kareer said. “You just wear a pair of glasses, and you go do things. It doesn’t need to come from the robot. It should come from something more scalable and passively generated, which is us.”

Kareer acquired a pair of Meta’s Project Aria research glasses, which contain a rich sensor suite and can record video from a first-person perspective through external RGB and SLAM cameras.

Kareer recorded himself folding a shirt while wearing the glasses and repeated the process. He did the same with other tasks such as placing a toy in a bowl and groceries into a bag. Then, he constructed a humanoid robot with pincers for hands and attached the glasses to the top to mimic a first-person viewpoint.

The robot performed each task repeatedly for two hours. Kareer said building a traditional training algorithm would take days of teleoperating and recording robot sensory data. For his project, he only needed to gather a baseline of sensory data to ensure performance improvement. 

Kareer bridged the gap between the two training sets with the EgoMimic algorithm. The robot’s task performance rating increased by as much as 400% among various tasks with just 90 minutes of recorded footage. It also showed the ability to perform these tasks in unseen environments.

If enough people wear Aria glasses or other smart glasses while performing daily tasks, it can create the passive data bank needed to train robots on a massive scale.

This type of data collection can enable nearly endless possibilities for roboticists to help humans achieve more in their everyday lives. Humanoid robots can be produced and trained at an industrial level and be able to perform tasks the same way humans do.

“This work is most applicable to jobs that you can get a humanoid robot to do,” Kareer said. “In whatever industry we are allowed to collect egocentric data, we can develop humanoid robots.”

Kareer will present his paper on EgoMimic at the 2025 IEEE Engineers’ International Conference on Robotics and Automation (ICRA), which will take place from May 19 to 23 in Atlanta. The paper was co-authored by Xu and School of IC Assistant Professor Judy Hoffman, fellow Tech students Dhruv Patel, Ryan Punamiya, Pranay Mathur, and Shuo Cheng, and Chen Wang, a Ph.D. student at Stanford.

Associate Professor Josiah Hester was one of 400 people awarded the Presidential Early Career Award for Scientists and Engineers (PECASE), the Biden Administration announced in a press release on Tuesday.

The PECASE winners’ research projects are funded by government organizations, including the National Science Foundation (NSF), the National Institutes of Health (NIH), the Centers for Disease Control and Prevention (CDC), and NASA. They will be invited to visit the White House later this year.

Hester joins Associate Professor Juan-Pablo Correa-Baena from the School of Materials Science and Engineering as the two Tech faculty who received the honor.

Hester said his nomination was based on the NSF Faculty Early Career Development Program (CAREER) award he received in 2022 as an assistant professor at Northwestern University. He said the NSF submits its nominations to the White House for the PECASE awards, but researchers are not informed until the list of winners is announced.

“For me, I always thought this was an unachievable, unassailable type of thing because of the reputation of the folks in computing who’ve won previously,” Hester said. “It was always a far-reaching goal. I was shocked. It’s something you would never in a million years think you would win.”

Hester is known for pioneering research in a new subfield of sustainable computing dedicated to creating battery-free devices powered by solar energy, kinetic energy, and radio waves. He co-led a team that developed the first battery-free handheld gaming device.

Last year, Hester co-authored an article published in the Association of Computing Machinery’s in-house journal, the Communications of the ACM, in which he coined the term “Internet of Battery-less Things.”

The Internet of Things is the network of physical computing devices capable of connecting to the internet and exchanging data. However, these devices eventually die. Landfills are overflowing with billions of them and their toxic power cells, harming our ecosystem.

In his CAREER award, Hester outlined projects that would work toward replacing the most used computing devices with sustainable, battery-free alternatives.

“I want everything to be an Internet of Batteryless Things — computational devices that could last forever,” Hester said. “I outlined a bunch of different ways that you could do that from the computer engineering side and a little bit from the human-computer interaction side. They all had a unifying theme of making computing more sustainable and climate-friendly.”

Hester is also a Sloan Research Fellow, an honor he received in 2022. In 2021, Popular Sciene named him to its Brilliant 10 list. He also received the Most Promising Engineer or Scientist Award from the American Indian Science Engineering Society, which recognizes significant contributions from the indigenous peoples of North America and the Pacific Islands in STEM disciplines.

President Bill Clinton established PECASE in 1996. The White House press release recognizes exceptional scientists and engineers who demonstrate leadership early in their careers and present innovative and far-reaching developments in science and technology.

More degrees to earn. More opportunities to teach. More lives to change.

He held six internships at companies such as Amazon, Google, and Intel Corporation, and each time he couldn’t wait to return to Georgia Tech’s campus.

His experiences at Georgia Tech have made it clear: Education is where he belongs.

“Every time I’ve interned, I didn’t like it, so I came back to school,” Rogers said. “Being in school for this long has never felt like compromising something else I would rather have been doing.”

Rogers said he’ll walk across the stage Thursday at McCamish Pavilion with no regrets as he receives his Ph.D. in computer science (CS) — the fourth degree he’s earned since arriving at Georgia Tech in 2011. He also holds his bachelor’s in computer engineering, a master’s in electrical and computer engineering, and a master’s in human-computer interaction (HCI).

That first master’s degree was mandated by his mother, Joan Dennis. She worked as a single parent without a college education in a competitive field in which most people had a master’s. 

The second master’s changed his life. Rogers planned to pursue an engineering-based Ph.D. after his first master’s, but he missed the application deadline. He looked for alternatives to searching for industry jobs, and he learned the application deadline for master’s programs was later than Ph.D. programs.

“It was a blessing in disguise,” Rogers said. “My background before the second master’s had been in computer engineering. It wasn’t people-focused, and I realized I cared more about people than electrons. Doing my master’s in HCI, I learned what it meant to do research with people in mind and how to design technology with people in mind.”

That decision put his research on a new trajectory. When he earned his master’s in human-computer interaction, he knew the Ph.D. he wanted to pursue. Accepted into the CS Ph.D. program, Rogers worked with former School of Interactive Computing professor and chair Ayanna Howard, who is now the Dean of the College of Engineering at Ohio State. 

Howard still advises Rogers along with School of IC associate professor Sonia Chernova. Rogers found a niche research field within human-robot interaction and built his dissertation around the ethics of robots and artificial intelligence and whether there are acceptable situations for a robot to lie to humans. For example, what should a chatbot tell a child if it is asked whether Santa Claus is real?

In 2023, Rogers became a finalist in Georgia Tech’s Three Minute (3M) Thesis Competition in which graduate students compete to explain their research in three minutes. He successfully defended his dissertation in November. 

Student Teacher

Rogers hasn’t lost touch with the new waves of incoming students over the years. Thousands of current students and Georgia Tech alumna know him as an instructor of the Computing for Engineers course (CS 1371), a CS course required for engineering majors.

It’s the same class Rogers took his first semester as a freshman, and it became one of his favorite undergraduate courses. A master’s degree is required to teach the course. He inquired about becoming an instructor when he knew he would return for a second master’s.

Rogers remembered the first day he taught in front of hundreds of students as his best and worst day at Georgia Tech. He taught the class in the morning, and later that day, he learned his mother unexpectedly passed away.

“It was a very conflicting time for me,” Rogers said. “Being able to teach the class helped me get through my mom’s death. I poured everything into it and tried to do everything I could to help students and be selfless the way my mom was toward me and my sister.”

Rogers said he wanted the class to be more than a requirement for engineering students to learn the basics of coding and computer programming. He saw it as an opportunity for engineering students to think differently about CS. He said some students have told him they switched their majors to CS because they took his course.

“I get to be the first exposure a lot of students get to computer science,” he said. “This class has 700 to 1,000 students every semester, and being able to have that kind of impact is very enticing.

“There’s never been a time since I’ve been teaching it when I didn’t look forward to it. Every day, I wake up excited to teach.”

Even when pursuing his Ph.D. consumed much of his time, he saw teaching as an outlet rather than a hindrance.

“Multiple people have told me to stop teaching because it doesn’t get you a Ph.D. For me, teaching has always been the fun part. There’s more in life than research, and teaching was an important counterbalance.”

Staying Connected

Rogers has also never been one to stay in a comfort zone or cut himself off from campus life. In addition to teaching CS 1371, Rogers has lived on campus throughout his time at Georgia Tech. As a grad student, he has been a resident advisor at Smith Hall and Hanson Hall, which house first-year students.

“I’m up to date on all the slang that comes out,” Rogers said. “It helps keep me relatable. I know what it’s like being a freshman taking this class, not knowing college, not knowing yourself, being confused. They’ll be going through problems in their lives, and I’m able to help them because I’ve been through some of the same things.”

Rogers said his career goal is to become a university president, but what's next in the immediate future is still up in the air. 

He’s applied for postdoc positions and hasn’t ruled out returning to Georgia Tech in that capacity. He may also teach CS 1371 one more semester in the spring while he sorts out his plans. However, he’s treating this semester as his last and preparing his goodbyes. 

“I don’t know what emotions I’ll feel,” Rogers said about attending the Ph.D. graduation ceremony Thursday. “I’ll let myself feel whatever I want. Throughout this process, I’ve been delusionally proud of myself for everything I’ve done.”

Two years since OpenAI introduced ChatGPT, dozens of lawsuits have been filed alleging technology companies have infringed copyright by using published works to train artificial intelligence (AI) models.

Academic AI research efforts could be significantly hindered if courts rule in the plaintiffs' favor. 

Desai and Riedl are Georgia Tech researchers raising awareness about how these court rulings could force academic researchers to construct new AI models with limited training data. The two collaborated on a benchmark academic paper that examines the landscape of the ethical issues surrounding AI and copyright in industry and academic spaces.

“There are scenarios where courts may overreact to having a book corpus on your computer, and you didn’t pay for it,” Riedl said. “If you trained a model for an academic paper, as my students often do, that’s not a problem right now. The courts could deem training is not fair use. That would have huge implications for academia.

“We want academics to be free to do their research without fear of repercussions in the marketplace because they’re not competing in the marketplace,” Riedl said. 

Desai is the Sue and John Stanton Professor of Business Law and Ethics at the Scheller College of Business. He researches how business interests and new technology shape privacy, intellectual property, and competition law. Riedl is a professor at the College of Computing’s School of Interactive Computing, researching human-centered AI, generative AI, explainable AI, and gaming AI. 

Their paper, Between Copyright and Computer Science: The Law and Ethics of Generative AI, was published in the Northwestern Journal of Technology and Intellectual Property on Monday.

Desai and Riedl say they want to offer solutions that balance the interests of various stakeholders. But that requires compromise from all sides.

Researchers should accept they may have to pay for the data they use to train AI models. Content creators, on the other hand, should receive compensation, but they may need to accept less money to ensure data remains affordable for academic researchers to acquire.

Who Benefits?

The doctrine of fair use is at the center of every copyright debate. According to the U.S. Copyright Office, fair use permits the unlicensed use of copyright-protected works in certain circumstances, such as distributing information for the public good, including teaching and research.

Fair use is often challenged when one or more parties profit from published works without compensating the authors.

Any original published content, including a personal website on the internet, is protected by copyright. However, copyrighted material is republished on websites or posted on social media innumerable times every day without the consent of the original authors. 

In most cases, it’s unlikely copyright violators gained financially from their infringement.

But Desai said business-to-business cases are different. The New York Times is one of many daily newspapers and media companies that have sued OpenAI for using its content as training data. Microsoft is also a defendant in The New York Times’ suit because it invested billions of dollars into OpenAI’s development of AI tools like ChatGPT.

“You can take a copyrighted photo and put it in your Twitter post or whatever you want,” Desai said. “That’s probably annoying to the owner. Economically, they probably wanted to be paid. But that’s not business to business. What’s happening with Open AI and The New York Times is business to business. That’s big money.”

OpenAI started as a nonprofit dedicated to the safe development of artificial general intelligence (AGI) — AI that, in theory, can rival human thinking and possess autonomy.

These AI models would require massive amounts of data and expensive supercomputers to process that data. OpenAI could not raise enough money to afford such resources, so it created a for-profit arm controlled by its parent nonprofit.

Desai, Riedl, and many others argue that OpenAI ceased its research mission for the public good and began developing consumer products. 

“If you’re doing basic research that you’re not releasing to the world, it doesn’t matter if every so often it plagiarizes The New York Times,” Riedl said. “No one is economically benefitting from that. When they became a for-profit and produced a product, now they were making money from plagiarized text.”

OpenAI’s for-profit arm is valued at $80 billion, but content creators have not received a dime since the company has scraped massive amounts of copyrighted material as training data.

The New York Times has posted warnings on its sites that its content cannot be used to train AI models. Many other websites offer a robot.txt file that contains instructions for bots about which pages can and cannot be accessed. 

Neither of these measures are legally binding and are often ignored.

Solutions

Desai and Riedl offer a few options for companies to show good faith in rectifying the situation.

• Spend the money. Desai says Open AI and Microsoft could have afforded its training data and avoided the hassle of legal consequences.
  
  “If you do the math on the costs to buy the books and copy them, they could have paid for them,” he said. “It would’ve been a multi-million dollar investment, but they’re a multi-billion dollar company.”
   
• Be selective. Models can be trained on randomly selected texts from published works, allowing the model to understand the writing style without plagiarizing. 
  
  “I don’t need the entire text of War and Peace,” Desai said. “To capture the way authors express themselves, I might only need a hundred pages. I’ve also reduced the chance that my model will cough up entire texts.”
   
• Leverage libraries. The authors agree libraries could serve as an ideal middle ground as a place to store published works and compensate authors for access to those works, though the amount may be less than desired.
  
  “Most of the objections you could raise are taken care of,” Desai said. “They are legitimate access copies that are secure. You get access to only as much as you need. Libraries at universities have already become schools of information.”

Desai and Riedl hope the legal action taken by publications like The New York Times will send a message to companies that develop AI tools to pump the breaks. If they don’t, researchers uninterested in profit could pay the steepest price.

The authors say it’s not a new problem but is reaching a boiling point.

“In the history of copyright, there are ways that society has dealt with the problem of compensating creators and technology that copies or reduces your ability to extract money from your creation,” Desai said. “We wanted to point out there’s a way to get there.”

A new paper authored by researchers in Georgia Tech's School of Interactive Computing reveals these models have trouble understanding contextual nuances that are specific to non-Western cultures.

Ph.D. student Tarek Naous and his advisors, associate professors Wei Xu and Alan Ritter, challenged ChatGPT-4 and an Arabic-specific LLM to choose the most appropriate word to complete a sentence. Some of the words it could choose from were contextually correct and would make sense within Arabic culture, while others fell within Western paradigms.

In questions asking for suggestions for food dishes, drinks, or names of Arabic women, the models chose Western responses — ravioli for food, whiskey for drinks, and Roseanne for names.

The implication is that LLMs appear to fall short in their ability to assist users who have non-Western backgrounds.

As a method of measuring cultural bias, the team also introduced CAMeL (Cultural Appropriateness Measure Set for LMs). CAMeL is a benchmark data set that includes 628 naturally occurring prompts and 20,368 entities spanning eight categories that contrast Arab and Western cultures.

Since the researchers announced their paper, it has received attention on social media and in external media.

To learn more about the authors and their work, read the article spotlighting them on VentureBeat.

A study by Georgia Tech and Stanford researchers compared the transcribing performance of leading ASR models for people using Standard American English (SAE) and three minority dialects — African American Vernacular English (AAVE), Spanglish, and Chicano English.

Interactive Computing Ph.D. student Camille Harris is the lead author of a paper accepted into the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) this week in Miami.

Harris recruited people who spoke each dialect and had them read from a Spotify podcast dataset, which includes podcast audio and metadata. Harris then used three ASR models — wav2vec 2.0, HUBERT, and Whisper — to transcribe the audio and compare their performances.

For each model, Harris found SAE transcription significantly outperformed each minority dialect. The models more accurately transcribed men who spoke SAE than women who spoke SAE. Members who spoke Spanglish and Chicano English had the least accurate transcriptions out of the test groups. 

While the models transcribed SAE-speaking women less accurately than their male counterparts, that did not hold true across minority dialects. Minority men had the most inaccurate transcriptions of all demographics in the study.

“I think people would expect if women generally perform worse and minority dialects perform worse, then the combination of the two must also perform worse,” Harris said. “That’s not what we observed. 

“Sometimes minority dialect women performed better than Standard American English. We found a consistent pattern that men of color, particularly Black and Latino men, could be at the highest risk for these performance errors.”

Addressing underrepresentation

Harris said the cause of that outcome starts with the training data used to build these models. Model performance reflected the underrepresentation of minority dialects in the data sets.

AAVE performed best under the Whisper model, which Harris said had the most inclusive training data of minority dialects.

Harris also looked at whether her findings mirrored existing systems of oppression. Black men have high incarceration rates and are one of the people groups most targeted by police. Harris said there could be a correlation between that and the low rate of Black men enrolled in universities, which leads to less representation in technology spaces.

“Minority men performing worse than minority women doesn’t necessarily mean minority men are more oppressed,” she said. “They may be less represented than minority women in computing and the professional sector that develops these AI systems.”

Harris also had to be cautious of a few variables among AAVE, including code-switching and various regional subdialects.

Harris noted in her study there were cases of code-switching to SAE. Speakers who code-switched performed better than speakers who did not. 

Harris also tried to include different regional speakers.

“It’s interesting from a linguistic and history perspective if you look at migration patterns of Black folks — perhaps people moving from a southern state to a northern state over time creates different linguistic variations,” she said. “There are also generational variations in that older Black Americans may speak differently from younger folks. I think the variation was well represented in our data. We wanted to be sure to include that for robustness.”

TikTok barriers

Harris said she built her study on a paper she authored that examined user-design barriers and biases faced by Black content creators on TikTok. She presented that paper at the Association of Computing Machinery’s (ACM) 2023 Conference on Computer Supported Cooperative Works. 

Those content creators depended on TikTok for a significant portion of their income. When providing captions for videos grew in popularity, those creators noticed the ASR tool built into the app inaccurately transcribed them. That forced the creators to manually input their captions, while SAE speakers could use the ASR feature to their benefit.

“Minority users of these technologies will have to be more aware and keep in mind that they’ll probably have to do a lot more customization because things won’t be tailored to them,” Harris said.

Harris said there are ways that designers of ASR tools could work toward being more inclusive of minority dialects, but cultural challenges could arise.

“It could be difficult to collect more minority speech data, and you have to consider consent with that,” she said. “Developers need to be more community-engaged to think about the implications of their models and whether it’s something the community would find helpful.”

Rachel Lowy, a Ph.D. student in the School of Interactive Computing, piloted a new human-computer interaction design course for IDD students in Georgia Tech’s Excel program. Excel is an Inclusive Postsecondary Education (IPSE) program that offers a four-year track for IDD students to earn two separate certificates.

Lowy said the course differs from typical technology courses taught to IDD students. It provides autonomy and encourages students to contribute input on how the course is designed and which technology projects they want to create. They reflect critically on the role of technology in the world and use that reflection to design technology for themselves.

The course is also unique because it involves a mix of professional educators and technology researchers working together. Lowy taught the class alongside her advisor, Assistant Professor Jennifer Kim, her lab colleague, Kaely Hall, master’s students in the Georgia Tech MS-HCI program, computer science undergraduates, and Excel educators.

“We have a few models of students designing next to designers in classrooms, but they tend to be only taught by professionals in K-12 education, not necessarily HCI researchers in higher education. They rarely include students with IDD,” she said. 

“In higher education, HCI projects may not go further than the classroom space. This course was special because we can teach these students with IDD high-level concepts about HCI and adopt their ideas into ongoing projects. We can keep working on them after the class has finished.”

Lowy said she designed the course based on previous work on accessible co-design and consulted with Assistant Professor Jessica Roberts, an educational technology researcher in the School of IC, to develop course materials. She refined the course with her co-teachers as she taught it, responding to observations and reflections from students. 

If the students had not been allowed to provide their input, Lowy and her team would never have learned how IDD students prefer to use different technologies. Lowy said they took that feedback to implement strength-based activities. 

“So much technology design for people with disabilities focuses on what they cannot do,” she said. “Our lab likes to focus on what they can do and their strengths.”

During one class, the researchers brought a robot dog into the classroom to determine whether it could supply emotional support to the students. The feedback they received showed the students were more interested in how the robot dog could be a companion in day-to-day activities.

“We came in with an idea of how the participants might want to use the technology,” Lowy said. “The students had a much broader view of what they might like to use this technology for. They reflected on their lives, and that’s exactly what we want good design to do.”

Lowy said she hopes the course serves as a blueprint for inclusive advanced technology courses at the university level.

“Most of their technology courses focus on workplace education like how to use Microsoft Suite, Google Calendar, or Outlook,” she said. “We’re working on more of a foundational level about how those technologies are designed and whether they work for them.”

She also said the course could be a step toward more inclusiveness in university classroom environments with traditional students and students with IDD learning together.

“Something that IPSE students have told me is that it’s hard to keep up with lectures, and they sometimes struggle to keep up in class,” she said. “It’d be great if they take a class specifically targeted to them at their own pace with a hands-on element to it, and they got to learn through experiential activities. Then they take the knowledge they’ve gleaned into an inclusive class where they work with their peers.”

She also suggested other models universities might offer, like an Intro to HCI course for IDD students that allows them to work on projects with students enrolled in the traditional Intro to HCI course.

“Any university with an IPSE program and an HCI program on campus can do this,” she said.

However, a study led by Georgia Tech and University of California, Berkeley researchers shows that annotations can lead people to draw different conclusions from the same visualizations. Their findings suggest readers should look beyond the presented data to make informed decisions.

“People question things less if they see something that’s visualized, and they think this is a reliable, trustworthy source they can use to form an opinion or persuade others,” said Cindy Xiong, an assistant professor in the School of Interactive Computing. “People don’t realize the persuasive power of visualization, and they’re not as vigilant to critically think about the data they interact with.”

For example, people tend to trust the information in an election data visualization. That makes them susceptible to narratives that visualization designers may use to obtain a certain outcome.

Working with Chase Stokes, a Ph.D. candidate at UC Berkeley’s School of Information, Xiong investigated how text position, semantic content, and biased wording impact viewers’ perception of visualizations.

They found people often reach the same conclusions suggested by titles and annotations on a chart.

“Visual changes have a great deal of impact on how people interpret a chart,” Stokes said. “Titles, captions, and annotations strongly affect people’s conclusions.”

Xiong and Stokes created a study centered around two hypothetical political parties — a blue party and a green party. They used a bar chart to show how many votes each party has received over the past three years. The data shows the blue party received more votes year after year than the green party, but the gap has closed each year.

The researchers surveyed participants to predict which party would win in the fourth year. Responses were split nearly 50-50 before leveraging highlights and annotations.

SKYSCENES compiles more than 33,000 annotated computer-generated aerial images. With applications in urban planning, disaster response, and autonomous navigation, the dataset trains computer vision models to better detect and identify objects in aerial images, which can be challenging for existing AI models.

Read the full story to learn how School of Interactive Computing Ph.D. student Sahil Khose and Assistant Professor Judy Hoffman developed this groundbreaking dataset to pave the way for the future of autonomous aviation.

While the tool proved effective, Hung began considering ways to deter plagiarism and improve the education system.

Plagiarism can be prevalent in online exams, so Hung looked at oral examinations commonly used in European education systems and rooted in the Socratic method.

One of the advantages of oral assessments is they naturally hinder cheating. Consulting ChatGPT wouldn’t benefit a student unless the student memorizes the entire answer. Even then, follow-up questions would reveal a lack of genuine understanding.

Hung drew inspiration from the 2009 reboot of Star Trek, particularly the opening scene in which a young Spock provides oral answers to questions prompted by AI.

“I think we can do something similar,” Hung said. “Research has shown that oral assessment improves people’s material understanding, critical thinking, and communication skills. 

“The problem is that it’s not scalable with human teachers. A professor may have 600 students. Even with teaching assistants, it’s not practical to conduct oral assessments. But with AI, it’s now possible.”

Hung developed The Socratic Mind with Starner, Scheller College of Business Assistant Professor Eunhee Sohn, and researchers from the Georgia Tech Center for 21st Century Universities (C21U).

The Socratic Mind is a scalable, AI-powered oral assessment platform leveraging Socratic questioning to challenge students to explain, justify, and defend their answers to showcase their understanding.

“We believe that if you truly understand something, you should be able to explain it,” Hung said. 

“There is a deeper need for fostering genuine understanding and cultivating high-order thinking skills. I wanted to promote an education paradigm in which critical thinking, material understanding, and communication skills play integral roles and are at the forefront of our education.”

Hung entered his project into the Learning Engineering Tools Competition, one of the largest education technology competitions in the world. Hung and his collaborators were among five teams that won a Catalyst Award and received a $50,000 prize.

Benefits for Students

The Socratic Mind will be piloted in several classes this semester with about 2,000 students participating. One of those classes is the Intro to Computing (CS1301) class taught by College of Computing Professor David Joyner.

Hung said The Socratic Mind will be a resource students can use to prepare to defend their dissertation or to teach a class if they choose to pursue a Ph.D. Anyone struggling with public speaking or preparing for job interviews will find the tool helpful. 

“Many users are interested in AI roleplay to practice real-world conversations,” he said. “The AI can roleplay a manager if you want to discuss a promotion. It can roleplay as an interviewer if you have a job interview. There are a lot of uses for oral assessment platforms where you can practice talking with an AI.

“I hope this tool helps students find their education more valuable and help them become better citizens, workers, entrepreneurs, or whoever they want to be in the future.”

Hung said the chatbot is not only conversational but also adverse to human persuasion because it follows the Socratic method of asking follow-up questions.

“ChatGPT and most other large language models are trained as helpful, harmless assistants,” he said. “If you argue with it and hold your position strong enough, you can coerce it to agree. We don’t want that.

“The Socratic Mind AI will follow up with you in real-time about what you just said, so it’s not a one-way conversation. It’s interactive and engaging and mimics human communication well.”

Educational Overhaul

C21U Director of Research in Education Innovation Jonna Lee and C21U Research Scientist Meryem Soylu will measure The Socratic Mind’s effectiveness during the pilot and determine its scalability.

“I thought it would be interesting to develop this further from a learning engineering perspective because it’s about systematic problem solving, and we want to create scalable solutions with technologies,” Lee said.

“I hope we can find actionable insights about how this AI tool can help transform classroom learning and assessment practices compared to traditional methods. We see the potential for personalized learning for various student populations, including non-traditional lifetime learners."

Hung said The Socratic Mind has the potential to revolutionize the U.S. education system depending on how the system chooses to incorporate AI.  

Recognizing the advancement of AI is likely an unstoppable trend. Hung advocates leveraging AI to enhance learning and unlock human potential rather than focusing on restrictions.

“We are in an era in which information is abundant, but wisdom is scarce,” Hung said. “Shallow and rapid interactions drive social media, for example. We think it’s a golden time to elevate people’s critical thinking and communication skills.”

For more information about The Socratic Mind and to try a demo, visit the project's website.

Terracell is roughly the size of a paperback book and uses microbes found in soil to generate energy for low-power applications. 

Previous designs for soil microbial fuel cells required water submergence or saturated soil. Terracell can function in soil with a volumetric water content of 42%

Terracell placed in Fast Company’s list of the best sustainability-focused designs of 2024.

Researchers at Northwestern University lead the multi-institution research team that designed Terracell.

Josiah Hester, an associate professor in Georgia Tech's School of Interactive Computing who previously worked at Northwestern, directs the Ka Moamoa Lab, where the project was conceived. 

The team includes researchers from Northwestern, Georgia Tech, Stanford, the University of California-San Diego, and the University of California-Santa Cruz.

Their research was published in January in the Proceedings of the Association for Computing Machinery on Interactive, Mobile, Wearable, and Ubiquitous Technologies. The researchers will also present this work at the ACM international joint conference on Pervasive and Ubiquitous Computing (Ubicomp), Oct. 5-9.

According to the Fast Company website, the Innovation by Design Awards recognize “designers and businesses solving the most crucial problems of today and anticipating the pressing issues of tomorrow.” Winners are published in Fast Company Magazine and are honored at the Fast Company Innovation Festival in the fall.

“Terracell could reduce e-waste and extend the useful lifetime of electronics deployed for agriculture, environmental monitoring, and smart cities,” Hester said. “We were honored to be recognized for the design innovation award. It is a testament to the promise of sustainable computing and our hope for a more sustainable world.”

For more information about Terracell, see the story featured on Northwestern Now, or visit the project’s website.

Proper instructions can be the difference between success and failure, whether for a parent putting together a crib or someone administering CPR.

While large language models (LLMs) can provide step-by-step instructions for assembling a crib, administering CPR, and other activities, Bolin Lai thinks they can go further.

Lai is a machine learning Ph.D. student who developed LEGO. This new framework allows generative artificial intelligence (AI) models to create first-person synthetic images based on text prompts. These images provide users with visual step-by-step instructions to complete a task.

For example, someone may not know how to properly handwash laundry if they’ve always relied on a washing machine. 

Lai said they could consult an LLM, but it provides instructions only in textual output. Users may feel better about doing the task correctly if they have a corresponding image to reference.

“Those instructions from LLMs could be very generic, so you’re reading lots of words, and it may not apply to your current situation,” Lai said. “Though you can input an image to GPT for more customized guidance, reading pure textual response isn’t efficient. Our model can understand the image and provide instructions by generating an image action frame showing people how to do it exactly.”

If a person wanted to know how to scrub a pair of trousers properly with a brush, they would first take a first-person photo of their situation. They can then upload that photo and prompt LEGO for instructions on washing the trousers with a brush. 

Based on the text in the prompt and the provided photo, the model generates a new image of someone scrubbing the trousers with the brush in the same environment.

The possibilities are innumerable, but Lai said his goal is to provide a way for people to learn new skills in everyday scenarios. Some of those skills could prove to be lifesaving.

“In some rural areas, there may not be any quick medical service available,” he said. “If an emergency happens, people can use this tool and find professional steps to assist the person who needs medical care.”

Lai started this project while interning at Meta GenAI and authored a paper titled LEGO: Learning Egocentric Action Frame Generation via Visual Instruction Tuning. His paper will be presented at the European Conference on Computer Vision Oct. 5-9 in Milan, Italy.

Gathering Data

Lai said his work stems from Meta’s release of the EGO4D dataset, a benchmark dataset consisting of first-person videos of humans performing everyday activities. The dataset was created to facilitate research in augmented and virtual reality and robotics.

Lai used still images from EGO4D to generate accurate and believable images in LEGO’s output.

“It’s so valuable, and they have corresponding annotations for the narration about what people are doing in the videos,” he said of EGO4D. “With so many egocentric videos, we can do much research on egocentric vision. We can have better data to train models and explore more action categories. We can learn the interaction of hands and objects and how the object’s state can change, such as moving from one place to another or changing its shape.”

Lai also curated images from a dataset called EPIC-KITCHENS, which contains first-person images of kitchen items, to bolster training.

Using a pair of smart glasses that could capture first-person images wherever he went, Lai then collected images of real-world scenarios that may require instructional assistance. He fed the images of those scenarios into LEGO and received accurate and believable synthetic images of completed tasks. 

He found that the model needs a single image to generate new images demonstrating a step-by-step process to complete a task.

“We show the model can a have high-quality generation of a real-world image. The task is challenging because the background in the user’s input image may be complex and chaotic. Other generative models are trained on all synthetic images with clean backgrounds and a few objects dominating the foreground. They oversimplify the problem and may not apply to the real world.”

From Images to Video

Lai envisions scaling his work to AI-generated video in which instructional videos could be the output instead of still images. These videos would show images of the instructional process and could be accompanied by narration.

He said that possibility is a long way off. Current generative AI video tools such as OpenAI’s Sora can generate videos up to 60 seconds long, but Lai says he doesn’t have access to the resources to reach that length. 

“We need more powerful computing resources to make it into a video, which was our initial goal, but we have found it difficult. It’s currently unaffordable for us, so we simplified the problem into image generation.”

These are details autonomous flying vehicles would need to know to function safely. However, few aerial image datasets exist that can adequately train the computer vision algorithms that would pilot these vehicles.

That’s why Georgia Tech researchers created a new benchmark dataset of computer-generated aerial images.

Judy Hoffman, an assistant professor in Georgia Tech’s School of Interactive Computing, worked with students in her lab to create SKYSCENES. The dataset contains over 33,000 aerial images of cities curated from a computer simulation program.

Hoffman said sufficient training datasets could unlock the potential of autonomous flying vehicles. Constructing those datasets is a challenge the computer vision research community has been working for years to overcome.

“You can’t crowdsource it the same way you would standard internet images,” Hoffman said. “Trying to collect it manually would be very slow and expensive — akin to what the self-driving industry is doing driving around vehicles, but now you’re talking about drones flying around. 

“We must fix those problems to have models that work reliably and safely for flying vehicles.”

Many existing datasets aren’t annotated well enough for algorithms to distinguish objects in the image. For example, the algorithms may not recognize the surface of a building from the surface of a street.

Working with Hoffman, Ph.D. student Sahil Khose tried a new approach — constructing a synthetic image data set from a ground-view, open-source simulator known as CARLA.

CARLA was originally designed to provide ground-view simulation for self-driving vehicles. It creates an open-world virtual reality that allows users to drive around in computer-generated cities.

Khose and his collaborators adjusted CARLA’s interface to support aerial views that mimic views one might get from unmanned aerial vehicles (UAVs). 

What's the Forecast?

The team also created new virtual scenarios to mimic the real world by accounting for changes in weather, times of day, various altitudes, and population per city. The algorithms will struggle to recognize the objects in the frame consistently unless those details are incorporated into the training data.

“CARLA’s flexibility offers a wide range of environmental configurations, and we take several important considerations into account while curating SKYSCENES images from CARLA,” Khose said. “Those include strategies for obtaining diverse synthetic data, embedding real-world irregularities, avoiding correlated images, addressing skewed class representations, and reproducing precise viewpoints.”

SKYSCENES is not the largest dataset of aerial images to be released, but a paper co-authored by Khose shows that it performs better than existing models. 

Khose said models trained on this dataset exhibit strong generalization to real-world scenarios, and integration with real-world data enhances their performance. The dataset also controls variability, which is essential to perform various tasks.

“This dataset drives advancements in multi-view learning, domain adaptation, and multimodal approaches, with major implications for applications like urban planning, disaster response, and autonomous drone navigation,” Khose said. “We hope to bridge the gap for synthetic-to-real adaptation and generalization for aerial images.”

Seeing the Whole Picture

For algorithms, generalization is the ability to perform tasks based on new data that expands beyond the specific examples on which they were trained.

“If you have 200 images, and you train a model on those images, they’ll do well at recognizing what you want them to recognize in that closed-world initial setting,” Hoffman said. “But if we were to take aerial vehicles and fly them around cities at various times of the day or in other weather conditions, they would start to fail.”

That’s why Khose designed algorithms to enhance the quality of the curated images.

“These images are captured from 100 meters above ground, which means the objects appear small and are challenging to recognize,” he said. “We focused on developing algorithms specifically designed to address this.”

Those algorithms elevate the ability of ML models to recognize small objects, improving their performance in navigating new environments.

“Our annotations help the models capture a more comprehensive understanding of the entire scene — where the roads are, where the buildings are, and know they are buildings and not just an obstacle in the way,” Hoffman said. “It gives a richer set of information when planning a flight.

“To work safely, many autonomous flight plans might require a map given to them beforehand. If you have successful vision systems that understand exactly what the obstacles in the real world are, you could navigate in previously unseen environments.”

For more information about Georgia Tech Research at ECCV 2024, click here.

Upol Ehsan is the first Georgia Tech graduate selected by BKC. As a fellow, he will contribute to its mission of exploring and understanding cyberspace, focusing on AI, social media, and university discourse.

Entering its 25th year, the BKC Harvard fellowship program addresses pressing issues and produces impactful research that influences academia and public policy. It offers a global perspective, a vibrant intellectual community, and significant funding and resources that attract top scholars and leaders.

The program is highly competitive and sought after by early career candidates and veteran academic and industry professionals. Cohorts hail from numerous backgrounds, including law, computer science, sociology, political science, neuroscience, philosophy, and media studies. 

“Having the opportunity to join such a talented group of people and working with them is a treat,” Ehsan said. “I’m looking forward to adding to the prismatic network of BKC Harvard and learning from the cohesively diverse community.”

While at Georgia Tech, Ehsan expanded the field of explainable AI (XAI) and pioneered a subcategory he labeled human-centered explainable AI (HCXAI). Several of his papers introduced novel and foundational concepts into that subcategory of XAI.

Ehsan works with Professor Mark Riedl in the School of Interactive Computing and the Human-centered AI and Entertainment Intelligence Lab.

Ehsan says he will continue to work on research he introduced in his 2022 paper The Algorithmic Imprint, which shows how the potential harm from algorithms can linger even after an algorithm is no longer used. His research has informed the United Nations’ algorithmic reparations policies and has been incorporated into the National Institute of Standards and Technology AI Risk Management Framework.

“It’s a massive honor to receive this recognition of my work,” Ehsan said. “The Algorithmic Imprint remains a globally applicable Responsible AI concept developed entirely from the Global South. This recognition is dedicated to the participants who made this work possible. I want to take their stories even further."

While at BKC Harvard, Ehsan will develop a taxonomy of potentially harmful AI effects after a model is no longer used. He will also design a process to anticipate these effects and create interventions. He said his work addresses an “accountability blindspot” in responsible AI, which tends to focus on potential harmful effects created during AI deployment.

A new large-language model (LLM) developed by Georgia Tech researchers can help them filter content that could risk their privacy and offer alternative phrasing that keeps the context of their posts intact.

According to a new paper that will be presented at the 2024 Association for Computing Linguistics(ACL) conference, social media users should tread carefully about the information they self-disclose in their posts.

Many people use social media to express their feelings about their experiences without realizing the risks to their privacy. For example, a person revealing their gender identity or sexual orientation may be subject to doxing and harassment from outside parties. 

Others want to express their opinions without their employers or families knowing.

Ph.D. student Yao Dou and associate professors Alan Ritter and Wei Xu originally set out to study user awareness of self-disclosure privacy risks on Reddit. Working with anonymous users, they created an LLM to detect at-risk content.

While the study boosted user awareness of the personal information they revealed, many called for an intervention. They asked the researchers for assistance to rewrite their posts so they didn’t have to be concerned about privacy.

The researchers revamped the model to suggest alternative phrases that reduce the risk of privacy invasion.

One user disclosed, “I’m 16F I think I want to be a bi M.” The new tool offered alternative phrases such as:

• “I am exploring my sexual identity.”
• “I have a desire to explore new options.”
• “I am attracted to the idea of exploring different gender identities.”

Dou said the challenge is making sure the model provides suggestions that don’t change or distort the desired context of the post.

“That’s why instead of providing one suggestion, we provide three suggestions that are different from each other, and we allow the user to choose which one they want,” Dou said. “In some cases, the discourse information is important to the post, and in that case, they can choose what to abstract.”

WEIGHING THE RISKS

The researchers sampled 10,000 Reddit posts from a pool of 4 million that met their search criteria. They annotated those posts and created 19 categories of self-disclosures, including age, sexual orientation, gender, race or nationality, and location.

From there, they worked with Reddit users to test the effectiveness and accuracy of their model, with 82% giving positive feedback.

However, a contingent thought the model was “oversensitive,” highlighting content they did not believe posed a risk. 

Ultimately, the researchers say users must decide what they will post. 

“It’s a personal decision,” Ritter said. “People need to look at this and think about what they’re writing and decide between this tradeoff of what benefits they are getting from sharing information versus what privacy risks are associated with that.”

Xu acknowledged that future work on the project should include a metric that gives users a better idea of what types of content are more at risk than others.

“It’s kind of the way passwords work,” she said. “Years ago, they never told you your password strength, and now there’s a bar telling you how good your password is. Then you realize you need to add a special character and capitalize some letters, and that’s become a standard. This is telling the public how they can protect themselves. The risk isn’t zero, but it helps them think about it.”

WHAT ARE THE CONSEQUENCES?

While doxing and harassment are the most likely consequences of posting sensitive personal information, especially for those who belong to minority groups, the researchers say users have other privacy concerns.

Users should know that when they draft posts on a site, their input can be extracted by the site’s application programming interface (API). If that site has a data breach, a user’s personal information could fall into unwanted hands.

“I think we should have a path toward having everything work locally on the user’s computer, so it doesn’t rely on any external APIs to send this data off their local machine,” Ritter said.

Ritter added that users could also be targets of popular scams like phishing without ever knowing it. 

“People trying targeted phishing attacks can learn personal information about people online that might help them craft more customized attacks that could make users vulnerable,” he said.

The safest way to avoid a breach of privacy is to stay off social media. But Xu said that’s impractical as there are resources and support these sites can provide that users may not get from anywhere else.

“We want people who may be afraid of social media to use it and feel safe when they post,” she said. “Maybe the best way to get an answer to a question is to ask online, but some people don’t feel comfortable doing that, so a tool like this can make them more comfortable sharing without much risk.”

For more information about Georgia Tech research at ACL, please visit https://sites.gatech.edu/research/acl-2024/.

Students returning to campus for a new semester often struggle to find time for physical activities because of their new routines and schedules. However, a new app developed at Georgia Tech helps busy students prioritize physical activity in their daily routines.

Ph.D. student Kefan Xu of the Ubicomp Health and Wellness Lab at Georgia Tech created Plannergy, a time management app that identifies open time blocks in users’ schedules. 

Xu introduced Plannergy at the Conference on Human Factors in Computing (CHI) in Honolulu, Hawaii in May. He says the app is ideal for college students because they tend to have busy and inconsistent schedules.

Plannergy allows users to track their schedules, reflect on what activities would be beneficial and timely, and strategize how to implement the activity into their schedule.

"Currently, the app is catered to people who’ve been physically inactive and have inconsistent schedules,” Xu said. “College students know their schedule will change when they begin a new semester. They need to get some physical activity and find opportunities in the day they can leverage. It could be as simple as walking to school instead of taking a scooter.”

Xu tested his app on 16 college students who planned their physical activities every seven days and followed a reflective iteration framework to track improvement. The results showed that Plannergy is an effective behavior change tool. The findings also indicate that it increases participants’ awareness of their schedules.

The American Heart Association says adults can reduce the risk of heart disease by participating in at least 150 minutes of moderate-intensity physical activity weekly.

The Centers for Disease Control and Prevention released a report in 2023 that found 72% of Americans aren’t meeting that standard.

As Xu points out in his paper, studies have shown that incorporating physical activity into a person’s routine usually helps them maintain it. However, he’s identified two common problems:

• People lack understanding about their schedules and routines.
• People have schedules that fluctuate from one day to the next.

“Individuals face a lot of changes in their life,” Xu said. “Maybe they’re a student who has graduated, and they’re going into industry, which means their daily routine will be different from what it was while they were in school. This app allows them to experiment with different time slots and activity types to figure out another way and help them update their activity routine no matter what life changes they face.”

CUSTOM FIT

Some users who have been inactive for extended periods may be unsure how much exercise they need. Plannergy can also help them determine the intensity level of the activity to help avoid overexertion. 

“If someone has been inactive for months, it’s hard to ask them to run two miles daily,” Xu said. “There’s much for them to figure out. How much do they want to do, and at what intensity level? This app lets them gradually figure out the ideal activity. They can continue to track their progress and see if improvements are needed.”

Plannergy is not limited to physical activity. Xu says one of the students in his study who worked out daily used the app to identify times in her schedule to take breaks or focus on more spiritual disciplines.

“She added yoga and removed some high-intensity physical activities, and her sleeping routine also changed,” Xu said.

Xu is working to improve the app. Future versions will have sensing technology to leverage health informatics so users can make better decisions. He also wants the app to record user data and make customized suggestions for activities that fit the user’s schedule and preferred exercise intensity level.

“The app requires manual tracking, which can create user burden,” he said. “I think in the future, the process could be more automated. We want to keep it flexible but add more scaffolding to enhance user experience.”

Ph.D. students Yasmine Belghith and Atefeh Mahdavi co-led a study in a museum setting that observed how middle schoolers interact with the popular AI chatbot ChatGPT. 

“It’s important for museums, especially science museums, to start incorporating these kinds of exhibits about AI and about using AI so the general population can have that avenue to interact with it and transfer that knowledge to everyday tools,” Belghith said.

Belghith and Mahdavi conducted their study with nine focus groups of 24 students at Chicago’s Museum of Science and Industry. The team used the findings to inform their design of AI exhibits that the museum could display as early as 2025. 

Belghith is a Ph.D. student in human-centered computing. Her advisor is Assistant Professor Jessica Roberts in the School of Interactive Computing. Magerko advises Mahdavi, a Ph.D. student in digital media in the School of Literature, Media, and Communication.

Belghith and Mahdavi presented a paper about their study in May at the Association for Computing Machinery (ACM) 2024 Conference on Human Factors in Computing Systems (CHI) in Honolulu, Hawaii.

Their work is part of a National Science Foundation (NSF) grant dedicated to fostering AI literacy among middle schoolers in informal environments.

Expanding Accessibility

While there are existing efforts to reach students in the classroom, the researchers believe AI education is most accessible in informal learning environments like museums.

“There’s a need today for everybody to have some sort of AI literacy,” Belghith said. “Many middle schoolers will not be taking computer science courses or pursuing computer science careers, so there needs to be interventions to teach them what they should know about AI.”

The researchers found that most of the middle schoolers interacted with ChatGPT to either test its knowledge by prompting it to answer questions or socialize with it by having human-like conversations. 

Others fit the mold of “content explorers.” They did not engage with the AI aspect of ChatGPT and focused more on the content it produced.

Mahdavi said regardless of their approach, students would get “tunnel vision” in their interactions instead of exploring more of the AI’s capabilities.

“If they go in a certain direction, they will continue to explore that,” Mahdavi said. “One thing we can learn from this is to nudge kids and show them there are other things you can do with AI tools or get them to think about it another way.”

The researchers also paid attention to what was missing in the students’ responses, which Mahdavi said was just as important as what they did talk about.

“None of them mentioned anything about ethics or what could be problematic about AI,” she said. “That told us there’s something they aren’t thinking about but should be. We take that into account as we think about future exhibits.”

Making an Impact

The researchers visited the Museum of Science and Industry June 1-2 to conduct the first trial run of three AI-based exhibits they’ve created. One of them is LuminAI, which was developed in Magerko’s Expressive Machinery Lab.

LuminAI is an interactive art installation that allows people to engage in collaborative movement with an AI dance partner. Georgia Tech and Kennesaw State recently held the first performance of AI avatars dancing with human partners in front of a live audience.

Duri Long, a former Georgia Tech Ph.D. student who is now an assistant professor at Northwestern University, designed the second exhibit. KnowledgeNet is an interactive tabletop exhibit in which visitors build semantic networks by adding different characteristics to characters that interact together.

The third exhibit, Data Bites, prompts users to build datasets of pizzas and sandwiches. Their selections train a machine-learning classifier in real time.

Belghith said the exhibits fostered conversations about AI between parents and children.

“The exhibit prototypes successfully engaged children in creative activities,” she said. “Many parents had to pull their kids away to continue their museum tour because the kids wanted more time to try different creations or dance moves.”

Assistant Professor Christopher MacLellan thinks teachable artificial intelligence (AI) agents are uniquely suited for this role and make ideal teammates for video gamers.

With the help of funding from the U.S. Department of Defense, MacLellan hopes to prove his theory with a conversational, task-performing agent he co-engineered called the Verbal Apprentice Learner (VAL).

“You need the ability to adapt to what your teammates are doing to be an effective teammate,” MacLellan said. “We’re exploring this capability for AI agents in the context of video games.” 

Unlike generative AI chatbots like ChatGPT, VAL uses an interactive task-learning approach. 

“VAL learns how you do things in the way you want them done,” MacLellan said. “When you tell it to do something, it will do it the way you taught it instead of some generic random way from the internet.”

A key difference between VAL and a chatbot is that VAL can perceive and act within the gaming world. A chatbot, like ChatGPT, only perceives and acts within the chat dialog.

MacLellan immersed VAL into an open-sourced, simplified version of the popular Nintendo cooperative video game Overcooked to discover how well the agent can function as a teammate. In Overcooked, up to four players work together to prepare dishes in a kitchen while earning points for every completed order.

How Fast Can Val Learn?

In a study with 12 participants, MacLellan found that users could often correctly teach VAL new tasks with only a few examples.

First, the user must teach VAL how to play the game. Knowing that a single human error could compromise results, MacLellan designed three precautionary features:

• When VAL receives a command such as "cook an onion," it asks clarifying questions to understand and confirm its task. As VAL continues to learn, clarification prompts decrease.
• An “undo” button to ensure users can reverse an errant command.
• VAL contains GPT subcomponents to interpret user input, allowing it to adapt to ambiguous commands and typos. The GPT subcomponents drive changes in VAL’s task knowledge, which it uses to perform tasks without additional guidance.

The participants in MacLellan’s study used these features to ensure VAL learned the tasks correctly. 

The high volume of prompts creates a more tedious experience. Still, MacLellan said it provides detailed data on system performance and user experience. That insight should make designing a more seamless experience in future versions of VAL possible.

The prompts also require the AI to be explainable.

“When VAL learns something, it uses the language model to label each node in the task knowledge graph that the system constructs,” MacLellan said. “You can see what it learned and how it breaks tasks down into actions.”

Beyond Gaming

MacLellan’s Teachable AI Lab is devoted to developing AI that inexperienced users can train.

“We are trying to come up with a more usable system where anyone, including people with limited expertise, could come in and interact with the agent and be able to teach it within just five minutes of interacting with it for the first time,” he said.

His work caught the attention of the Department of Defense, which awarded MacLellan multiple grants to fund several of his projects, including VAL. The possibilities of how the DoD could use VAL, on and off the battlefield, are innumerable.

“(The DoD) envisions a future in which people and AI agents jointly work together to solve problems,” MacLellan said. “You need the ability to adapt to what your teammates are doing to be an effective teammate.

“We look at the dynamics of different teaming circumstances and consider what are the right ways to team AI agents with people. The key hypothesis for our project is agents that can learn on the fly and adapt to their users will make better teammates than those that are pre-trained like GPT.”

Design Your Own Agent

MacLellan is co-organizing a gaming agent design competition sponsored by the Institute of Electrical and Electronic Engineers (IEEE) 2024 Conference on Games in Milan, Italy.

The Dice Adventure Competition invites participants to design their own AI agent to play a multi-player, turn-based dungeon crawling game or to play the game as a human teammate. The competition this month and in July offers $1,000 in prizes for players and agent developers in the top three teams.

What began as an internship at NASA turned into a unique opportunity for fourth-year Ph.D. student Adam Coscia. Coscia worked under the supervision of an interdisciplinary team of collaborative researchers from Caltech, the Jet Propulsion Laboratory (JPL) Caltech manages for NASA and the ArtCenter College of Design.

Coscia’s mentors recommended him to a Caltech research team led by Victoria Orphan, a renowned microbial ecologist who studies microbial communities in the ocean and how they function within habitats in deep seafloor sediments. 

Orphan and her team, the Orphan Lab at Caltech, have conducted their research since 2004. They recently decided to take a data visualization approach to record their findings and plan future expeditions.

“Historically, our data sets have been discrete and have lived in separate Excel spreadsheets,” Orphan said. “Maybe at the end, we’ll do some statistical analysis to find correlations in that data. Then we compare those to our maps. We didn’t have a way of consolidating everything under one umbrella that allows us to learn more about these ecosystems.”

Orphan said her team typically takes one or two research expeditions off the California coast annually. They spend three weeks using remotely operated vehicles (ROVs) to collect sediment samples from the ocean floor. Because time is at a premium, identifying the locations of the best samples is crucial.

Orphan is also an adjunct scientist at the Monterey Bay Aquarium Research Institute (MBARI) and works with the Seafloor Mapping Lab. The lab uses an ROV-mounted low-altitude survey system to produce detailed maps of seafloor topography. 

To help the Orphan Lab work effectively with topographic and photographic data, Coscia designed DeepSee, an interactive web browser that can annotate and chart data using 3D visualization models and environmental maps.

“The idea is once you have the samples, and you’re interested in a specific area with prior samples, you can go in and annotate on the map where to collect samples next with our drawing tool,” Coscia said.

“We focused on the exploration and notetaking process with maps and data and having new ways of visualizing it. Scientists can draw and map out all their samples in real time. They can reference specific data much easier and determine where the team should go to get the best samples.”

The Orphan Lab has taken DeepSee live onboard its ship for its two most recent expeditions. Orphan has noticed an increased efficiency in expedition planning.

“The infrastructure put in place by Adam will make this an enabling tool not only for my group but for other oceanographers and scientists in other fields — anywhere there is a spatial distribution of information you want to connect to other metadata,” she said.

Orphan brings new researchers into her lab at Caltech every year, and DeepSee has accelerated the process of getting newcomers up to speed.

“We can onboard them much easier and give them a sense of what data is available and where we’ve collected information in a way that’s much clearer than having them refer to an Excel spreadsheet,” she said.

DeepSee also creates 3D data models under the sea floor using data interpolation, which estimates new data points based on the range of a set of known data points. Using the known data points, DeepSee fills in the blanks of the estimated data quality the researchers may find in nearby locations or further underneath the surface where samples were collected.

“You would never see anything visually below the sea floor,” Coscia said. “You’d have to go dig. But our 3D models show you that you might have data suggesting a hotspot just a few feet below the floor. That tells you where to sample next.”

Coscia aims to incorporate machine learning (ML) models into a future version of DeepSee that will use collected data to predict future sites for sampling. However, ML model accuracy requires significantly more data.

Coscia hopes the current version of the tool catches on so researchers can more easily incorporate machine learning into their work.

For now, the current version has plenty of uses, he said.

“Being able to organize and see your data, especially with maps, is always valuable,” he said. “My passion is helping researchers and scientists see their data in new and valuable ways.”

Coscia authored a paper on developing DeepSee, which he presented in May at the Conference on Human Factors in Computing Systems (CHI) in Honolulu, Hawaii.

Jain is the creator of VCoder, a new tool that enhances the visual perception capabilities of multimodal large language models (MLLMs). Jain said MLLMs like GPT-4 with vision (GPT-4V) are prone to miss obscure objects that blend in with other objects in an image.

Jain paused his TV as he watched The One with the Halloween Party episode of the popular TV Series Friends. 

Chandler stood out the most in a pink bunny costume while holding hands with Ross in a potato costume. As the two prepared for an arm-wrestling match with Joey and Phoebe, multiple groups socialized behind them.

Jain wondered how accurate GPT-4V would be if he prompted itto describe everything happening in this image.

“I watch a lot of TV series, so I frequently think about ways to leverage or include some aspects of those into my work,” said Jain, a Ph.D. student in the School of Interactive Computing. “The scene was very cluttered, so I thought, what questions could I ask GPT-4 about this show.”

On the surface, the generative AI chatbot knew much about the image. It knew which show and episode it was from and recognized the man in the bunny costume as Chandler. It knew the main characters were hosting a Halloween party.

But when Jain prompted the chatbot to count the number of people in the image, he discovered that GPT-4V and its open-source counterparts fell short of performing the simplest task.

It answered 10 when the correct answer was 14. In the right corner of the image, there is a group of people standing in front of a dark curtain that GPT-4V had missed. 

AI Paradox

Jain had a theory — the MLLMs had not been trained for the object perception task and did not have the necessary information to perceive the objects in the foreground and background.

“We started testing it with different pictures, and GPT-4V kept underperforming,” Jain said. “The key takeaway is that it couldn’t do a simple task such as counting the people in the scene, but it knew complex information such as what was happening and who the characters were. This phenomenon is Moravec’s Paradox in Perception — the MLLMs visually reason about complex questions but fail at simple object perception tasks like counting.”

Jain said he has worked on image segmentation tools for the past two years. That includes when he was a research intern at Picsart AI under his now Ph.D. advisor Humphrey Shi, an associate professor in the School of Interactive Computing.

The core idea behind VCoder is to act as a perceptive eye for the MLLM, using segmentation and depth maps obtained through established computer vision frameworks with minimal training costs. The tool also proposes evaluation metrics for object perception tasks like counting and ordering.

Its training and evaluation set consists of images from Common Objects in Context (COCO), a widely used object perception dataset. Associate Professor James Hays from the School of Interactive Computing was one of the academic collaborators who worked with Microsoft to create COCO.

Improving MLLMs

Though VCoder didn’t know which show the image was from, it accurately described everything, including the number of people. It showed as much as 10% more accuracy than its nearest competitor.

It could also identify the order of objects in a scene.

Jain designed VCoder to integrate easily with existing MLLMs. He said augmenting MLLMs with VCoder leads to an MLLM with sound general reasoning and object perception capabilities.

However, he added he was unsure if integration would happen because companies like Open AI, which created GPT-4V, may overlook it.

“There’s no way to know if they will integrate since GPT-4V is a closed model, and their main motivation is to make a product useful to consumers in general,” he said. “They often ignore these small issues.”

Jain’s paper was accepted into the Institute of Electrical and Electronics Engineers’ 2024 Conference on Computer Vision and Pattern Recognition (CVPR), occurring June 17-21 in Seattle. CVPR is the highest-ranked conference in computer vision according to Google Scholar.

Nivedita Arora received the 2024 Association for Computing Machinery (ACM) Doctoral Dissertation Award during an awards ceremony on Saturday in San Francisco. Arora, an assistant professor at Northwestern University, is the first Georgia Tech alumna to win the award, which includes a prize of $20,000.

Arora was a postdoctoral researcher at Georgia Tech’s School of Interactive Computing during the 2022-2023 academic year. She also earned her Ph.D. in computer science and her master’s in human-computer interaction from Georgia Tech.

At Northwestern, she directs the VAK Sustainable Computing Lab, which re-envisions computing from a sustainability-first approach.

“The ACM Doctoral Dissertation Award is the most prestigious recognition for doctoral research in our field,” said Josiah Hester, an associate professor in the School of Interactive Computing who mentored Arora during her postdoc. “The award is a testament to the recipient's exceptional contributions to the field of computing, marking them as a world-class leader and innovator.”

Arora creates sustainable computational materials that harvest energy from their surrounding environments and can be responsibly disposed of at the end of their life cycles. Under the advisement of Professor Thad Starner and former Georgia Tech Professor Gregory Abowd, she won the dissertation award for her work involving interactive sticky notes.

The interactive sticky notes perform computing tasks and allow wireless communication without battery dependency. 

Through her dissertation, Sustainable Interactive Wireless Stickers: From Materials to Devices on Applications, Arora demonstrated that interactive sticky notes can capture audio, store it as memory, and relay it to another location. For example, an Amazon Alexa user can communicate commands to Alexa without being nearby.

“With rising climate change and e-waste, it is imperative to build computing technologies with a sustainability-first approach,” Arora said. “My dissertation represents this core thinking. I am honored that ACM has recognized my research on sustainable computational materials. I am extremely grateful to my advisers, collaborators, friends, and family for their support.”

Her dissertation also earned Outstanding Dissertation recognition from Georgia Tech’s College of Computing in 2023. She also won the college’s 2022 Outstanding Graduate Research Assistant Award.

Arora was a finalist in the 2022 Fast Company Design Innovation Competition. In 2021, She won the ACM Gaetano Borriello Outstanding Ubiquitous Computing Student Award and was named an EECS Rising Star and a Foley Scholar.

Since humans often associate objects in a home with the room they are in, Naoki Yokoyama thinks robots that navigate human environments to perform assistive tasks should mimic that reasoning.

Roboticists have employed natural language models to help robots mimic human reasoning over the past few years. However, Yokoyama, a Ph.D. student in robotics, said these models create a “bottleneck” that prevents agents from picking up on visual cues such as room type, size, décor, and lighting. 

Yokoyama presented a new framework for semantic reasoning at the Institute of Electrical and Electronic Engineers (IEEE) International Conference on Robotics and Automation (ICRA) last month in Yokohama, Japan. ICRA is the world’s largest robotics conference.

Yokoyama earned a best paper award in the Cognitive Robotics category with his Vision-Language Frontier Maps (VLFM) proposal.

Assistant Professor Sehoon Ha and Associate Professor Dhruv Batra from the School of Interactive Computing advised Yokoyama on the paper. Yokoyama authored the paper while interning at the Boston Dynamics’ AI Institute.

“I think the cognitive robotic category represents a significant portion of submissions to ICRA nowadays,” said Yokoyama, whose family is from Japan. “I’m grateful that our work is being recognized among the best in this field.”

Instead of natural language models, Yokoyama used a renowned vision-language model called BLIP-2 and tested it on a Boston Dynamics “Spot” robot in home and office environments.

“We rely on models that have been trained on vast amounts of data collected from the web,” Yokoyama said. “That allows us to use models with common sense reasoning and world knowledge. It’s not limited to a typical robot learning environment.”

What is Blip-2?

BLIP-2 matches images to text by assigning a score that evaluates how well the user input text describes the content of an image. The model removes the need for the robot to use object detectors and language models. 

Instead, the robot uses BLIP-2 to extract semantic values from RGB images with a text prompt that includes the target object. 

BLIP-2 then teaches the robot to recognize the room type, distinguishing the living room from the bathroom and the kitchen. The robot learns to associate certain objects with specific rooms where it will likely find them.

From here, the robot creates a value map to determine the most likely locations for a target object, Yokoyama said.

Yokoyama said this is a step forward for intelligent home assistive robots, enabling users to find objects — like missing keys — in their homes without knowing an item’s location. 

“If you’re looking for a pair of scissors, the robot can automatically figure out it should head to the kitchen or the office,” he said. “Even if the scissors are in an unusual place, it uses semantic reasoning to work through each room from most probable location to least likely.”

He added that the benefit of using a VLM instead of an object detector is that the robot will include visual cues in its reasoning.

“You can look at a room in an apartment, and there are so many things an object detector wouldn’t tell you about that room that would be informative,” he said. “You don’t want to limit yourself to a textual description or a list of object classes because you’re missing many semantic visual cues.”

While other VLMs exist, Yokoyama chose BLIP-2 because the model:

• Accepts any text length and isn’t limited to a small set of objects or categories.
• Allows the robot to be pre-trained on vast amounts of data collected from the internet.
• Has proven results that enable accurate image-to-text matching.

Home, Office, and Beyond

Yokoyama also tested the Spot robot to navigate a more challenging office environment. Office spaces tend to be more homogenous and harder to distinguish from one another than rooms in a home. 

“We showed a few cases in which the robot will still work,” Yokoyama said. “We tell it to find a microwave, and it searches for the kitchen. We tell it to find a potted plant, and it moves toward an area with windows because, based on what it knows from BLIP-2, that’s the most likely place to find the plant.”

Yokoyama said as VLM models continue to improve, so will robot navigation. The increase in the number of VLM models has caused robot navigation to steer away from traditional physical simulations.

“It shows how important it is to keep an eye on the work being done in computer vision and natural language processing for getting robots to perform tasks more efficiently,” he said. “The current research direction in robot learning is moving toward more intelligent and higher-level reasoning. These foundation models are going to play a key role in that.”

Top photo by Kevin Beasley/College of Computing.

The new curriculum introduced by Artificial Intelligence for Georgia (AI4GA) has taught middle school students to use and understand AI. It’s also equipped middle school teachers to teach the foundations of AI.

AI4GA is a branch of a larger initiative, the Artificial Intelligence for K-12 (AI4K12). Funded by the National Science Foundation and led by researchers from Carnegie Mellon University and the University of Florida, AI4K12 is developing national K-12 guidelines for AI education. 

Bryan Cox, the Kapor research fellow in Georgia Tech’s Constellation Center for Equity in Computing, drove a transformative computer science education initiative when he worked at the Georgia Department of Education. Though he is no longer with the DOE, he persuaded the principal investigators of AI4K12 to use Georgia as their testing ground. He became a lead principal investigator for AI4GA.

“We’re using AI4GA as a springboard to contextualize the need for AI literacy in populations that have the potential to be negatively impacted by AI agents,” Cox said.

Judith Uchidiuno, an assistant professor in Georgia Tech’s School of Interactive Computing, began working on the AI4K12 project as a post-doctoral researcher at Carnegie Mellon under lead PI Dave Touretzky. Joining the faculty at Georgia Tech enabled her to be an in-the-classroom researcher for AI4GA. She started her Play and Learn Lab at Georgia Tech and hired two research assistants devoted to AI4GA.

Focusing on students from underprivileged backgrounds in urban, suburban, and rural communities, Uchidiuno said her team has worked with over a dozen Atlanta-based schools to develop an AI curriculum. The results have been promising.

“Over the past three years, over 1,500 students have learned AI due to the work we’re doing with teachers,” Uchidiuno said. “We are empowering teachers through AI. They now know they have the expertise to teach this curriculum.”

AI4GA is in its final semester of NSF funding, and the researchers have made their curriculum and teacher training publicly available. The principal investigators from Carnegie Mellon and the University of Florida will use the curriculum as a baseline for AI4K12.

STARTING STUDENTS YOUNG

Though AI is a complex subject, the researchers argue middle schoolers aren’t too young to learn about how it works and the social implications that come with it.

“Kids are interacting with it whether people like it or not,” Uchidiuno said. “Many of them already have smart devices. Some children have parents with smart cars. More and more students are using ChatGPT. 

“They don’t have much understanding of the impact or the implications of using AI, especially data and privacy. If we want to prepare students who will one day build these technologies, we need to start them young and at least give them some critical thinking skills.”

Will Gelder, a master’s student in Uchidiuno’s lab, helped analyze data exploring the benefits of co-designing the teaching curriculum with teachers based on months of working with students and learning how they understand AI. Rebecca Yu, a research scientist in Uchidiuno’s lab, collected data to determine which parts of the curriculum were effective or ineffective.

Through the BridgeUP STEM Program at Georgia Tech, Uchidiuno worked with high school students to design video games that demonstrate their knowledge of AI based on the AI4GA curriculum. Students designed the games using various maker materials in 2D and 3D representations, and the games are currently in various stages of development by student developers at the Play and Learn Lab.  

“The students love creative projects that let them express their creative thoughts,” Gelder said. “Students love the opportunity to break out markers or crayons and design their dream robot and whatever functions they can think of.”

Yu said her research shows that many students demonstrate the ability to understand advanced concepts of AI through these creative projects.

“To teach the concept of algorithms, we have students use crayons to draw different colors to mimic all the possibilities a computer is considering in its decision-making,” Yu said.

“Many other curricula like ours don’t go in-depth about the technical concepts, but AI4GA does. We show that with appropriate levels of scaffolding and instructions, they can learn them even without mathematical or programming backgrounds.” 

EMPOWERING TEACHERS

Cox cast a wide net to recruit middle school teachers with diverse student groups. A former student of his answered the call.

Amber Jones, a Georgia Tech alumna, taught at a school primarily consisting of Black and Latinx students. She taught a computer science course that covered building websites, using Excel, and basic coding.

Jones said many students didn’t understand the value and applications of what her course was teaching until she transitioned to the AI4GA curriculum.

“AI for Georgia curriculum felt like every other lesson tied right back to the general academics,” Jones said. “I could say, ‘Remember how you said you weren’t going to ever use y equals mx plus b? Well, every time you use Siri, she's running y equals mx plus b.’ I saw them drawing the connections and not only drawing them but looking for them.”

Connecting AI back to their other classes, favorite social media platforms, and digital devices helped students understand the concepts and fostered interest in the curriculum.

Jones’s participation in the program also propelled her career forward. She now works as a consultant teaching AI to middle school students. 

“I’m kind of niche in my experiences,” Jones said. “So, when someone says, ‘Hey, we also want to do something with a young population that involves computer science,’ I’m in a small pool of people that can be looked to for guidance.”

AI4GA quickly cultivated a new group of experts within a short timeframe.

“They’ve made their classes their own,” Cox said. “They add their own tweaks. Over the course of the project, the teachers were engaged in cultivating the lessons for their experience and their context based on the identity of their students.”

Gaurav Verma and Yuxi Wu are recipients of 2023 J.P. Morgan AI Research Ph.D. Fellowship Awards. They are among 13 scholars being honored this year by J.P. Morgan Chase & Co. for AI research projects taking on real-world challenges.

"Our goal is to recognize and enable the next generation of leading AI researchers. We want to create an environment where researchers can inspire change and make a lasting impact in our communities and across our industry," said Manuela Veloso, Ph.D., head of AI Research, J.P. Morgan Chase & Co.

Verma is pursuing his Ph.D. in the School of Computational Science and Engineering. Working with his advisor, Assistant Professor Srijan Kumar, Verma expects to ensure safety, equity, and well-being by creating multimodal learning and natural language processing approaches to achieve better human-AI interactions.

Wu is a Ph.D. candidate in the School of Interactive Computing. Empowering people regarding their privacy concerns is at the core of her research. Wu examines how cross-sector, collective action systems could better support end-user privacy. Professor Keith Edwards and Adjunct Assistant Professor Sauvik Das advise Wu.

"It's inspiring to see our students and their work being honored with these prestigious fellowships," said Irfan Essa, computer science professor and director of the Machine Learning Center at Georgia Tech.

"Georgia Tech continues to lead in AI education and research. These fellowships for Gaurav and Yuxi are evidence that we're continuing to move in the right direction."

Verma and Wu are part of a spectrum of AI research spanning Georgia Tech. To unite this broad community and ensure it continues moving in the right direction, the Institute recently established AI Hub at Georgia Tech.

"AI has a deep history at Georgia Tech, and we continue to serve as leaders in many areas of AI research and education," said Essa, interim co-director of AI Hub at Georgia Tech.

"Bringing all areas of AI under one umbrella, AI Hub at Georgia Tech will provide structure and governance as the Institute continues to lead and innovate in the burgeoning discipline of AI."

Eye tracking is an essential component of AR/VR systems, but current systems have some limitations. These include a large size due to bulkier lens-based cameras and the high communication cost between the camera and the backend system.

Georgia Tech School of Computer Science Associate Professor Yingyan (Celine) Lin, Ph.D. student Hoaran You, and postdoctoral student Yang (Katie) Zhao have developed a new eye-tracking system that works around these limitations by combining a recently developed lens-less camera, algorithm, and acceleration processor designs.

“The current VR headsets are too heavy, gaming can lag, and using the controller is cumbersome. Combined, this prevents users from having a truly immersive experience. We mitigate all these problems,” said You.

This new system, EyeCoD: An Accelerated Eye Tracking System via FlatCam-based Algorithm & Accelerator Co-Design, replaces the traditional camera lens with FlatCam, a lensless camera 5x – 10x thinner and lighter. Combined with FlatCam, the team’s system enables eye tracking to function at a reduced size, with improved efficiency, and without sacrificing the accuracy of the tracking algorithm. The system could also enhance user privacy by not including a lens-based camera.

Another feature of the EyeCoD system is that it only puts the portion of the screen that a user’s eyes focus on in high resolution. It does this by predicting where a user’s eyes may land, then instantaneously rendering these areas in high res. These computational savings, plus a dedicated accelerator, underpin EyeCoD’s ability to boost processing speeds and efficiency.   

The team received the Office of Technology Licensing’s Tech Ready Grant for its efforts earlier this year. Tech Ready Grants offer $25,000 to help faculty transition projects from the lab to the marketplace.  

The team hopes to use the funds to integrate the current demos into a compact eye-tracking system for use in commercial VR/AR headsets.

Along with winning the Tech Ready Grant, the team presented EyeCoD at the International Symposium on Computer Architecture (ISCA) 2022. IEEE Micro included the work in its Top Picks from the Computer Architecture Conferences for 2023. The annual publication highlights “significant research papers in computer architecture based on novelty and potential for long-term impact.”  

EyeCoD is a collaborative work. Collaborators include Rice University Professor Ashok Veeraraghavan, whose team provided the technical support and design of the FlatCam camera in EyeCoD; and Ziyun Li, of Meta, who provided technical inputs to ensure that the EyeCoD system aligns with industry AR/VR specifications.  

The Atlanta Interdisciplinary Artificial Intelligence (AIAI) Network brings together computing, humanities, and social justice researchers from Georgia Tech, Clark Atlanta University, Emory University, and community partner DataedX.

Carl DiSalvo, Georgia Tech School of Interactive Computing professor, is an AIAI co-principal investigator (co-PI). Andre Brock, an associate professor in the School of Literature, Media, and Communication serves on the network’s steering committee.

DiSalvo said the idea for the AIAI Network had been in the works for years. However, the researchers now have the needed funding thanks to the Mellon Foundation. The grant allows the network to hire its first graduate students for the 2023-2024 academic year.

“The Mellon grant provides resources that we didn’t have before,” DiSalvo said. “There are students doing work on topics related to AI, computing, humanities, and social justice. They were difficult to fund, but now there’s funding. This has a material impact on supporting graduate students and their research, and that impact is immediate.”

The Mellon award also provides seed money for the network to distribute grants to researchers in the Atlanta community. Brandeis Marshall, CEO of DataedX Group and co-PI, said the network wants to put Atlanta voices at the forefront of conversations about AI bias that aren’t limited to the academic community.

“We want people within Atlanta to connect with it, understand it, and be a part of it,” Marshall said. “We want small businesses and nonprofits to feel like they have a place within these conversations about tech. It’s for the everyday person, not just the academics.”

Lauren Klein, PI and Emory associate professor of English, said the AIAI Network offers a humanistic lens on controversial AI issues. She said it was important that each PI or steering committee member be open to research contributions from the humanities.

“The proposed technical solutions are not coming from people who have expertise in these issues of systemic racism, sexism, and structural oppression,” Klein said. “The people who have expertise with these issues and how they surface in AI are humanities scholars. We want to bring humanities researchers to the table with technical researchers as equal partners.”

Klein said the Mellon Foundation recognized the AIAI Network’s goals aligned with its recent commitment to funding projects centered on social justice.

“It aligns with the work that the Mellon Foundation is trying to do,” Klein said. “They’ve made social justice the top-level concern of all projects they fund. They don’t fund a lot of initiatives, so choosing to invest in us is meaningful.”

One of the biggest challenges the network will face is steering the conversation away from the prominent AI “doomer” narrative and toward existing AI bias. While the former is theorized, the latter continues to impact marginalized and minority communities.

“It’s not just what we are going to do today, but also how what we’re doing today will impact what we are trying to influence for tomorrow,” Marshall said. “How can we dampen the AI hype and AI doom narratives and promote AI reality?”

AIAI Network will take a multifaceted approach to promote a more realistic understand of AI. Some of the tactics the group will use include:

• Humanities-focused research projects
• Public design and education workshops
• Guest speaker series
• Courses taught by principal investigators and steering committee members
• Seed grants for Atlanta researchers doing like-minded work

“We’re going to educate ourselves, those in the community, and those aspiring to be in this field on how we can be more solution based and solution oriented,” Marshall said. “There are courses, research and research projects built on the framework of talking about AI bias in a productive way and not one focused on extreme stances around AI hype or AI doom.”

Even worse, they leave rejected applicants with no recourse.

People can have similar experiences when applying for jobs or petitioning their health insurance providers. While AI tools determine the fate of people in difficult situations daily, Upol Ehsan says more thought should be given to challenging these decisions or working around them.

Ehsan, a Georgia Tech explainable AI (XAI) researcher, says many rejection cases are not the applicant’s fault. Rather, it’s more likely a “seam” in the design process — a mismatch between what designers thought the AI could do and what happens in reality.

Ehsan said “seamless design” is the standard practice of AI designers. While the goal is to create a process by which users get what they need without interruption or barriers, seamless design has a way of doing just the opposite. 

No amount of thought or design input will keep AI tools from making mistakes. When mistakes happen, those impacted by them want to know why they happened.

Because seamless design often includes black-boxing — the act of concealing the AI’s reasoning — answers are never provided.

But what if there were a way to challenge an AI’s decisions and turn its mistakes into benefits for end users? Ehsan believes that can be done through “seamful design.”

n his latest paper, Seamful Explainable AI: Operationalizing Seamful Design in XAI, Ehsan proposes a strategic way of anticipating AI harms, learning their reasonings, and leveraging mistakes instead of concealing them. 

GIVING USERS MORE OPTIONS

In his research, Ehsan worked with loan officers who used automated lending support systems. The seams, or flaws, he discovered in these tools’ processes impacted applicants and lenders.

“The expectation is that the lending system works for everyone,” Ehsan said. “The reality is that it doesn’t. You’ve found the seam once you’ve figured out the difference between expectation and reality. Then we ask, ‘How can we show this to end users so they can leverage it?’”

To give users options when AI negatively impacts them, Ehsan suggests three things for designers to consider:

• Actionability: Does the information about the flaw help the user take informed actions on the AI’s recommendation?
• Contestability: Does the information provide the resources necessary to justify saying no to the AI?
• Appropriation: Does identifying these seams help the user to adapt and appropriate the AI’s output in a way that is different from the provided design but helps the user make the right decision?

Ehsan uses the example of someone who was rejected for a loan despite having a good credit history. The rejection may have been due to a seam, such as a flawed discriminating algorithm, in the AI that screens the applications.

A post-deployment process is needed in cases like this to mitigate damage and empower affected end users. Loan applicants, for instance, should be allowed to contest the AI’s decision based on known issues with an algorithm. 

AGAINST THE GRAIN

Ehsan said his idea for seamful design is outside of the mainstream vernacular. However, his challenge to current accepted principles is gaining traction.

He is now working with cybersecurity, healthcare, and sales companies that are adopting his process.

These companies may pioneer a new way of thinking in AI design. Ehsan believes this new mindset can allow designers to switch to a proactive mindset instead of being stuck in a reactive state of conducting damage control.

“You want to stay a little ahead of the curve so you’re not always caught off guard when things happen,” Ehsan said. “The more proactive you can be and the more passes you can take at your design process, the safer and more responsible your systems will be.”

Ehsan collaborated with researchers from Georgia Tech, the University of Maryland, and Microsoft. They will present their paper later this year at the 2024 Association for Computing Machinery’s Conference on Computer-Supported Cooperative Work and Social Computing (CSCW) in Costa Rica. 

“Seamful design embraces the imperfect reality of our world and makes the most out of it,” he said. “If it becomes mainstream, it can help us address the hype cycle AI suffers from now. We don’t need to overhype AI’s capacity or impose unachievable goals. That’d be a gamechanger in calibrating people’s trust in the system.” 

Alyssa Sheehan has seen firsthand how companies can struggle to leverage new technologies meant to improve systems and benefit workers. She collaborated with dozens of companies as the director of the Georgia Center of Innovation's aerospace team from 2022 to 2023.

That experience inspired the Ph.D. candidate and 2022 Foley Scholar to explore the effects on workers when technology is implemented to automate traditional paper-based processes. Making Meaning from the Digitalization of Blue-Collar Work won a best paper award at the 2023 Conference on Computer Supported Cooperative Work and Social Computing (CSCW) this week in Minneapolis.

“I’m trying to cast meaningful work into a new light with automation and technology design,” Sheehan said. “The intention is so focused on delivering efficiency and optimizing the process. Companies and technologists forget about user input from workers using these systems.”

Microsoft and other major tech companies have announced commitments to use technology to foster a culture of meaningful work within the workplace. However, Sheehan said that small businesses often lack the resources and knowledge required to incorporate such beneficial technology. Others design the technology with only productivity in mind and without considering if it makes their employees’ jobs more meaningful.

“There’s a lot of research that shows there’s a technology gap, particularly for small businesses,” Sheehan said. “I’m not always advocating for technology as a solution, but I look at what exists critically and ask, ‘Is this technology doing what we want it to? If the goal is to support workers, how is it doing that?’”

Sheehan worked with a small Georgia-based manufacturing company to conduct an 18-month study. She designed and deployed off-the-shelf tools to automate the company’s shipping and receiving processes that required time and paperwork.

With the support of researchers from Georgia Tech’s Institute of People and Technology (IPAT), she customized a wearable and mobile app. The workers used the app to check off critical tasks within the shipping process one by one.  

The results were mixed.

Sheehan said many ground-floor shipping experts were frustrated by the frequency of having to repack orders because of customer complaints about improper shipping. The workers insisted they’d done the job correctly. The mobile app allowed them to take pictures of each order after packaging for quality assurance.

The workers appreciated the feature, but they also provided negative feedback. In some cases, the app required workers to perform tasks contrary to methods that suited them and made them feel productive. It also took away a sense of autonomy and pride in expertise from workers because it instructed them what to do step by step. Instead of making the job easier, workers felt like their superiors didn’t trust them to do the job correctly.

“It helped in certain areas like not having to take notes on paper anymore and using outdated equipment. However, they struggled to see how it would preserve meaning in their job in terms of working with their hands and doing various tasks at any given time.

“We create universal systems and solutions for mobile apps that are often deployed without understanding the context of organizational practices. That’s a problem. Now, the workers have to adapt their processes to make this tool work in practice. They’re being asked to give up how they do things,” Sheehan said.

She added that automated technology systems need to go beyond convenience and productivity, and these systems may cause more harm than good if it diminishes meaning and value from workers.

“By leaving the worker perspective out of the design process, we limit the potential of these technologies,” she said. “Productivity still relies on people being engaged in the process. If we’re going to create true productivity, we need to make sure those jobs are valuable and that people feel what they do matters. That leads to less turnover and higher job satisfaction rates.”

With a handful of students and researchers at Georgia Tech looking to make breakthroughs in home robotics and object rearrangement, Ramachandruni searched for what others had overlooked.

“To an extent it was challenging, but it was also an opportunity to look at what people are already doing and to get more familiar with the literature,” said Ramachandruni, a Ph.D. student in Robotics. “(Associate) Professor (Sonia) Chernova helped me in deciding how to zone in on the problem and choose a unique perspective.”

Ramachandruni started exploring how a home robot might organize objects according to user preferences in a pantry or refrigerator without prior instructions required by existing frameworks.

His persistence paid off. The 2023 IEEE International Confrence on Robots and Systems (IROS) accepted Ramachandruni’s paper on a novel framework for a context-aware object rearrangement robot.

“Our goal is to build assistive robots that can perform these organizational tasks,” Ramachandruni said. “We want these assistive robots to model the user preferences for a better user experience. We don’t want the robot to come into someone’s home and be unaware of these preferences, rearrange their home in a different way, and cause the users to be distressed. At the same time, we don’t want to burden the user with explaining to the robot exactly how they want the robot to organize their home.”

Ramachandruni’s object rearrangement framework, Context-Aware Semantic Object Rearrangement (ConSOR), uses contextual clues from a pre-arranged environment within its environment to mimic how a person might arrange objects in their kitchen.

“If our ConSOR robot rearranged your fridge, it would first observe where objects are already placed to understand how you prefer to organize your fridge,” he said. “The robot then places new objects in a way that does not disrupt your organizational style.”

The only prior knowledge the robot needs is how to recognize certain objects such as a milk carton or a box of cereal. Ramachandruni said he pretrained the model on language datasets that map out objects hierarchically.

“The semantic knowledge database we use for training is a hierarchy of words similar to what you would see on a website such as Walmart, where objects are organized by shopping category,” he said. “We incorporate this commonsense knowledge about object categories to improve organizational performance.

“Embedding commonsense knowledge also means our robot can rearrange objects it hasn’t been trained on. Maybe it’s never seen a soft drink, but it generally knows what beverages are because it’s trained on another object that belongs to the beverage category.”

Ramachandruni tested ConSOR against two model training baselines. One used a score-based approach that learns how specific users group objects in an environment. It then uses the scores to organize objects for users. The other baseline used the GPT-3 large language model prompted with minimal demonstrations and without fine-tuning to determine the placement of new objects. ConSOR outperformed both baselines.

“GPT-3 was a baseline we were comparing against to see whether this huge body of common-sense knowledge can be used directly without any sort of frame,” Ramachandruni said. “The appeal of LLMs is you don’t need too much data; you just need a small data set to prompt it and give it an idea. We found the LLM did not have the correct inductive bias to correctly reason between different objects to perform this task.”

Ramachandruni said he anticipates there will be scenarios where user input is required. His future work on the project will include minimizing the effort required by the user in those scenarios to tell the robot its preferences.

“There are probably scenarios where it’s just easier to ask the user,” he said. “Let’s say the robot has multiple ideas of how to organize the home, and it’s having trouble deciding between them. Sometimes it’s just easier to ask the user to choose between the options. That would be a human-robot interaction addition to this framework.”

IROS is taking place this week in Detroit.

It’s an exciting time to be in the field of robotics, he said, and that’s why the new assistant professor in the School of Interactive Computing wants to be at Georgia Tech.

“The robotics faculty at Georgia Tech is particularly strong,” he said. “The reorganization and refocus of the Institute for Robotics and Intelligent Machines and the Machine Learning Ph.D. program is something that is also very special among our peer group universities.”

Garg described his research as unlocking “common sense” for robotics. In essence, he thinks carefully about mistakes that robots might make in the real world and how to preempt them before deployment.

He said building common sense into robots could be the difference over the next few years if robotics researchers are to reach a new pinnacle of achievements.

“Common sense reasoning in real-world robotics is the challenge of the next decade,” he said.
“Robots have a lot of requirements for common sense. Simple things like don’t pack glassware under heavy stuff. We should think about these problems holistically and not trying to build robots in isolation.”

Garg earned his master’s degree in industrial engineering from Georgia Tech in 2011. He also has a master’s in computer science and a doctorate in machine learning and robotics from the University of California-Berkley.

Garg spent the past four years teaching robotics, reinforcement learning, robot manipulation, and computer vision at the University of Toronto. He is also a senior research scientist at Nvidia, working on machine learning for robot manipulation.

What interested you about coming to Georgia Tech?

The primary draw was the research culture and the school’s strength, particularly in robotics and machine learning. What is also attractive about Georgia Tech is that it’s not just computer science and engineering that are strong. There is no shortage of collaborators outside of the computer science neighborhood if I want to pursue projects in climate sciences, material sciences, or statistics.

What will your research consist of?

My research will consist of three big-picture topics. First, foundation models for representation and reasoning. How should we talk about common sense and problem solving? The second is generative AI in the context of robots so we can know more about the world through the predictive sense, which allows for better planning. The third pillar would be reinforcement — the robot learning to do something within its own abilities.

What inspired you to pursue this field of research?

Getting robots to do stuff on command is one of the longstanding science fiction challenges, right? We have used science fiction examples for many decades, but the progress has been slow. The confluence of machine learning and the ability to reason gives us the tools to solve this problem.

In the next decade, we will see more progress in robot manipulation than in the last 40 to 50 years combined. A fundamental set of problems has been solved in the last five to seven years. In the next 10 to 12 years, we will see a boom in the percolation of this technology in everyday life.

What do you hope to accomplish here at Georgia Tech?

I want to train students to be leaders in this space for the next decade, whether they choose to be in academia or start new companies. The other thing I want to establish in the research ecosystem is a consortium of robotics researchers within Georgia Tech to work closely with industry. This will enable tech transfer so what we create at Georgia Tech can be brought to fruition in industry and make a broader impact.

What are you looking forward to about teaching your students and how do you plan to work with them?

I’ll be focusing on a Ph.D. course in reinforcement learning. There is no deep reinforcement learning course offered regularly to in-person students, so that is a need I hope to fill.

The other thing I’ve been developing is a course on robot learning. The idea is that this will be a hands-on course that enables people with little background in robotics to get up to speed of a professional robotics engineer within six months.

The CCRL, however, functions as a “library” that allows the robot to remember everything it has learned while performing the simple tasks. Each newly obtained skill is added to the library and leveraged for more complex skills. A turning motion, for instance, can be learned on top of walking while serving as the basis for navigation skills.

Kumar said CCRL has broken new ground on interactive navigation research. Interactive navigation is one of several navigation solutions that allow robots to navigate in the real world. These solutions include point navigation, which trains a robot to reach a point on a map, and object navigation, which teaches it to reach a selected object.

Interactive navigation requires a robot to reach a goal location while interacting with obstacles on the way, which has proven to be the most difficult for robots to learn.

The key, Kumar said, to get a robot to go from walking to pushing an object is in the joints and the robot discovering the different types of motions it can make with them.

So far, Kumar’s policy has reached 10 skills that a robot can learn and deploy. The number of skills it can learn on one policy depends on the hardware the programmer is using.

“It just takes longer to train as you keep adding more skills because now the policy also has to figure out how to incorporate all these skills in different situations,” he said. “But theoretically, you can keep adding more skills indefinitely as long as you have a powerful enough computer to run the policies.”

Kumar said he sees CCRL being useful for home assistant robots, which are required to be agile and limber to navigate around a cluttered household. He also said it could possibly serve as a guide dog for the visually impaired.

“If you have obstacles in front of someone who is visually impaired, the robot can just clear up the obstacles as the person is walking, open the door for them, and things like that,” he said.

The web-based app is being developed by a group of Georgia Tech Master of Science in Human-Computer Interaction (MS-HCI) students. It can inform tenants of their rights, help them to ensure landlords have properly followed the law and help them to better prepare for their court hearings.

The students have been developing the web app for the Atlanta Volunteer Lawyers Foundation (AVLF), which provides free legal services to residents in Fulton County, including advice for tenants facing evictions.

Jessie Chiu, one of the students who worked on the app before graduating in May, said the eviction problem throughout Atlanta is expansive.

In an article published by WABE, a local NPR affiliate, Fulton County Chief Magistrate Judge Cassandra Kirk said her office receives 40,000 eviction filings per year or about 800 per week.

“Atlanta has one of the highest eviction rates in America, Fulton County especially,” Chiu said. “It disproportionately affects people of marginalized communities. There’s a huge power imbalance between landlords and tenants, and landlords will employ attorneys who specifically work on evicting tenants, while tenants have limited legal resources and knowledge.”

The AVLF operates the Housing Court Assistance Center (HCAC), a walk-in advice clinic where volunteer attorneys and paralegals provide free legal advice to tenants facing eviction. The clinic is located at the Fulton County Magistrate Court in downtown Atlanta.

Working inside the Culture and Technology (CAT) Lab directed by Associate Professor Betsy DiSalvo, Chiu and fellow student Xiao Luo led the group’s research efforts this year just before graduating.

To gain firsthand insights into the legal process, they spent three months interviewing tenants and observing counseling sessions with volunteer HCAC attorneys. This helped them understand what tenants needed from the web app while ensuring the advice provided would be consistent with advice given by attorneys.

“It’s important to get in front of our end users,” Chiu said. “Once we got to sit in on these sessions with tenants undergoing eviction, we gained valuable insights into the complexity of the process. We decided to expand the available resources, covering each stage of the defense process so it became a comprehensive, step-by-step guide.”

Recognizing the limitations of the HCAC to address a city-wide problem, Luo voiced the need for a digital solution.

“There’s limited access to the attorneys,” Luo said. “At the clinic, there’s always a long queue of people there. This could provide users with immediate access to resources whenever they need them.”

One of the most challenging procedures in the eviction process is the seven-day response time given to tenants. Anyone served with an eviction notice must file an answer within that timeframe. Luo said the paperwork is long and complicated and can easily trip up tenants with limited legal knowledge.

Failure to submit the answer within seven days condemns tenants to a default judgment hearing, where judges often rule in favor of landlords.

Currently, the web app provides users with reminders to file their answers and check the court website for any scheduled hearings. Looking to the future, Luo said the goal is to incorporate real-time updates and notifications. The group will need the cooperation of the court system to do so. 

Luo said her team is dedicated to optimizing the web app’s functionality and simplifying the process, making it more efficient and less time-consuming for users.

“We hope to develop a more flexible solution where users won’t feel tied to a step-by-step process,” Luo said. “Instead, they can follow some simple prompts, describe their situations, and it will provide them with the solutions and advice they need.”

Chiu said she hopes the project is the first step toward revolutionizing the eviction defense landscape.

“There’s still a lot of work to do, but it’s exciting to see how tech can be used for leveling the playing field and giving people equal access to information and the tools they need to fight for their rights,” Chiu said.

Viraj Prabhu, a Ph.D. student in Georgia Tech’s School of Interactive Computing, introduced the LANCE (Language-Guided Counterfactuals) method in a recent research paper that shows how deep object recognition models are prone to taking shortcuts through context clues to produce images.

Ideally, models should understand exactly what they’re prompted to search for, Prabhu said, but because of spurious correlation, they tend to use irrelevant information in images as they make predictions.

Prabhu used LANCE to stress test well-known models that have been trained on the image database ImageNet. Working with Assistant Professor Judy Hoffman and co-authors Sriram Yenamandra and Prithvijit Chattopadhyay, he discovered many instances in which the models were overly reliant on context in the images they produced.

In some examples, the models showed they were using weather in the background to classify images rather than recognizing the object of interest.

On another stress test, Prabhu challenged the models to classify images with seatbelts. All the test images contained seatbelts inside cars. When Prabhu generated new images by changing the parameters to “seatbelts on a bus,” the performance and accuracy of the trained models dropped. This suggested the models thought seat belts were exclusive to cars.

“When a model is getting something right, is it getting it right because it really understands it, or is it picking up on some context clues and relying on them?” Prabhu said.

“There is no reason why it should be relying on what kind of vehicle it is to know whether there is a seatbelt, but models often do this. It’s more generally known as model bias or a spurious correlation problem.”

The models displayed the same flaws when Prabhu used LANCE to test images for dog sleds. The models almost exclusively associated dog sleds with Huskies, leading them to focus their searches on the breed most associated with sleds.

From left to right, Sriram Yenamandra, Viraj Prabhu, and Prithvijit Chattopadhyay, discuss their LANCE method for detecting input changes that deep object recognition models are sensitive to. Photos by Kevin Beasley/College of Computing.

Prabhu said the prompts given to the models were generated by finetuning LLaMA, a large-language model created by Meta AI, while using training data automatically generated by Open AI’s ChatGPT. For an image of someone riding a bike, he generated a caption using an automated captioning system. Then, he used the finetuned LLaMA to make a structured change to the caption, only changing a single concept at a time.

“It would change ‘person riding a bicycle’ to ‘person carrying a bicycle,’ and then we pass it to the generative model and use it to generate a new image while changing nothing else,” he said. “Using a recently introduced targeted editing technique from Google Research based on prompt-to-prompt tuning, we can now change only the relationship between the person and bicycle. Then we get an image of a person carrying a bicycle, with everything else being the same. Now we can use this as a counterfactual test image.”

That allows Prabhu to compare the model’s new prediction to the original. If the prediction has changed, it’s likely the model is relying on spurious correlations.

Prabhu said the LANCE method can be applied at scale for any new data set.

Spurious correlation has been a known weak link for deep learning models, but Prabhu said the benefit of LANCE is that it allows programmers to probe their models for those weaknesses before deployment.

Traditionally, these models are trained through goal-oriented methods in which the models receive points for displaying the correct image and lose points for getting them wrong. Prabhu said that’s the most likely reason why the artificial intelligence in the models tries to find shortcuts, like using contextual clues, to achieve their goals.

The implications also expand beyond diagnosing object recognition models trained on ImageNet. LANCE can be applied to computer vision technology used in self-driving vehicles, which need to be as foolproof as possible before they’re deployed on the road.

“In high-stakes applications like self-driving, people are using discriminative approaches — you have an object detection system that can detect cars and pedestrians and draw boxes around them,” Prabhu said. “Using LANCE, we can probe these discriminative models using generative approaches and make them better. The hope is we can discover failures before they happen.”

Her father was an engineer, and her mother was a psychologist. She was interested in both.

“I realized there was this whole field of human-robot interaction, which combines those two fields,” Botti said. “I get to code the robot and write algorithms and I also get to run human-subject experiments and analyze how people feel about the robot.”

It isn’t the dynamics and intricacies of robotics that drives Botti as much as the human element that HRI explores.

“When I tell people I work with robotics, they ask if I’m working on Terminator,” she said. “I’m trying to do the opposite — building robots that are helpful and customizable.”

Botti, a fourth-year Ph.D. student under the advisement of Interactive Computing Assistant Professor Matthew Gombolay, recently received the P.E.O. Sisterhood’s Scholar Award. The merit-based award is given to women pursuing doctoral-level degrees and comes with a $20,000 scholarship.

The P.E.O. Sisterhood is an educational organization founded in 1869 dedicated to the advancement of women in higher education with more than 6,000 local chapters in North America and about 250,000 active members.

“It’s nice to be honored,” Botti said. “I wasn’t really expecting it. It can help me go to more conferences that I wouldn’t necessarily be able to attend otherwise. A lot of my time here was during Covid, so we didn’t get to travel much. It’ll be nice to broaden my network and see other types of research.”

The bulk of Botti’s research has focused on training robots through human demonstration, also known as Learning from Demonstration (LfD).

Botti received a best paper award in 2022 from the International Conference on Human-Robot Interaction for co-authoring “MIND MELD: Personalized Meta-Learning for Robot-Centric Imitation Learning.” The paper explores robots designed to be taught by everyday people and how the robot can learn correctly if users lack the expertise to teach them.

“People can be suboptimal when giving demonstrations because they may not understand the robot or there may be a correspondence problem,” Botti said. “Maybe your arm is different from the robot’s arm, and when you perform the motion, it may not work as well. People can also take shortcuts that the robot should not follow. And people are heterogenous. The way I would show the robot to do something is different from the way someone else would.

“We developed an algorithm that learns from suboptimal and heterogenous demonstrators. It uses personalized embedding that describes how a person is suboptimal, and then we can use that embedding to learn how to correct their demonstrations and shift them to be better.”

Botti also co-authored a paper that was accepted to the 2022 Conference on Robotic Learning (CoRL), which expanded upon her research in the HRI paper as to how robots can provide feedback to their human trainers.

Botti is now researching how to develop in-home robot assistant for the older adults that can perform daily chores, such as loading a dishwasher, and adapt to user preferences.

“Erin is on the cusp of putting robots in the hands of regular people in their homes to learn and perform assistive tasks and validating decades of research in robotics,” Gombolay said. “Her unique focus on robotic assistance for the elderly will have significant broader impacts on society. She is bright, inquisitive, savvy, and fearless, and this award will help her leverage those assets to change the world of robotics.”

Researchers created a dataset called ChattyChef, which uses natural language processing models that can help a user cook a recipe. Using the open-source large language model GPT-J, ChattyChef’s dataset of cooking dialogues follows recipes with the user.

The researchers presented their AI in the paper, “Improved Instruction Ordering in Recipe-Grounded Conversation,” presented at the 61st annual meeting of the Association for Computational Linguistics.

Although other researchers have theorized about the possibility of an AI chef, Georgia Tech’s work pushes the field forward. “We are one of the first research teams to analyze the challenges of using large language models for building an AI chef,” said Duong Le, a Ph.D. student in the School of Interactive Computing.

Most attempts at using language models for cooking fail because GPT-J doesn’t understand what the user wants to do next, or user intent, and has difficultly tracking how far the user is in the recipe — what the researchers call the “state of the conversation.” It also can’t easily answer clarification questions, like about ingredient amounts or cooking times.

For example, maybe someone is trying to cook hashbrowns. The AI tells them to melt butter in the pan and add the potatoes. The user then asks about the next step. A bad bot might jumble the order and tell them to serve the hashbrown even though they haven’t finished cooking it yet. Or a user asks a follow-up question about how long to cook the hashbrown, and AI won’t be precise enough, instead giving a general time and not specifying the cooking time for each side.

With this in mind, the researchers ensured their model had two key features:

• User intent detection to determine the user’s current intent within a fixed set of possibilities, such as “Ask for next instruction” or “Ask for details about ingredients.”
• Instruction state tracking to identify which recipe step the user is on, which works with 80% accuracy.

The combined information from these features supports the third innovation of ChattyChef — response generation. User intent helps generate the best response to answer a user’s question. The instruction state selects the most relevant parts of the recipe rather than including the entire recipe, to avoid confusing the user or burdening them with extra steps as they are cooking.

The ChattyChef dataset is built off WikiHow recipes with positive ratings and fewer than eight steps. The researchers crowdsourced people to role play how they might use ChattyChef to determine what instructions would be best to include in the dataset.

The researchers believe the innovations of ChattyChef could be used in many domains besides cooking, such as repair manuals or software documentation.

Duong Minh Le, Ruohao Guo, Wei Xu, and Alan Ritter. 2023. Improved instruction ordering in recipe-grounded conversation. arXiv preprint arXiv:2305.17280.

This research is supported in part by the National Science Foundation awards IIS-2112633 and IIS-2052498.

The harmful effects of an algorithm can range from labeling and categorizing someone into a box that inaccurately depicts who they really are, to altering one’s future because of the way they answered a question on an exam or job application.

In some cases, algorithms can reinforce systems that are unjust or oppressive, argues Georgia Tech researcher and School of Interactive Computing Ph.D. candidate Upol Ehsan in his paper, The Algorithmic Imprint, which was presented at the 2022 Association for Computing Machinery’s FAcct Conference.

In 2020, Ehsan saw a news report about protests occurring in the United Kingdom. Students across the U.K. spoke out against the results of the General Certificate of Education (GCE) A-level examinations, which had been graded by an algorithm for the first time. The A-levels are the final exams taken before university in the U.K. and have a major impact on whether students can attend their desired institutions.

The Office of Qualifications and Examinations Regulation (Ofqual), the GCE exam governing body in the U.K., said the COVID-19 pandemic had made it necessary to pivot from manual grading to using an algorithm. Protests demonstrated that students found this change to be unacceptable, arguing the algorithm was biased against people from poorer economic backgrounds.

Ofqual removed the algorithm from its grading, but that didn’t solve the problem. Ehsan and his colleagues argue the effects of Ofqual’s algorithm lingered long after its removal. The situation is one example of how algorithms can leave hard imprints on the societies in which they are deployed.

“Most students we interviewed for the paper had their grades improved,” Ehsan said. “But they were still angry. That’s when I started thinking, ‘Why are people still angry even if their results aren’t bad?’”

And why was the U.K. the only country to receive any media coverage when the same exams are administered in more than 160 countries, including Ehsan’s home country of Bangladesh?

“This is not a U.K. issue,” he said. "This is a global issue. If we don’t share people’s stories, their narratives get erased from the historical narrative. If we didn’t bring this up, largely speaking, the Bangladesh narrative would’ve never been captured as the catastrophe that it was.

“Protests were also going on elsewhere. It’s just that they were never covered. These kids were protesting just as much as the U.K. kids.”

The fact that many of the other countries who use the GCEs are members of the Commonwealth — meaning they were once occupied by the British Empire — wasn’t lost on Ehsan.

The voices in the Global South weren’t being heard after suffering the effects of an algorithm designed by the Global North.

“Right now, the current way we evaluate an algorithm’s impact is from the birth to the death of the algorithm, from its deployment to its destruction,” Ehsan said. “When an algorithm is deployed, we do an impact assessment. When it is no longer deployed, we stop it, and that’s where we think this is the end.

“And that is the fundamental flaw in our thinking. Even when the algorithm was taken out, its harms persisted.”

The imprints can be even more damaging when they mimic or reinforce modern and historical systems of discrimination and oppression. Ehsan argues that was the case with the Commonwealth nations who also use the GCEs, and the exams were already considered to be unfair and biased before the algorithm was deployed.

“It feels like I’m paying money to my ex-colonizer for a piece of certificate that tells the world I am no dumber than a local UK kid,” said one student whom Ehsan interviewed during his research. “Sometimes it’s hard to ignore that reality.”

Ehsan compared the effects of colonialism to trying to erase pencil markings from a piece of paper. Even after the eraser has been used, the traces of the pencil markings are still visible.

“One of the things you see in postcolonialism is that there are remnants of the British infrastructure that we still live with today,” Ehsan says. “Just because colonizers leave, does not mean colonialism has left. Just because an algorithm has left, doesn’t mean its impact has left.”

Ehsan said the goal of his paper is to bring awareness to the imprints that algorithms can leave so developers can consider the potential impacts of an algorithm before deploying it.

“One of the things I wanted to do was drive policy changes,” he said. “I didn’t want this to be a research project just to have a research project. I had a moral reason behind it. I was driven by it.”

Ehsan also said he would like to see developers, researchers, and practitioners design algorithms in a way in which their impacts can be controlled and mitigated, and if an algorithm harms a group of people, reparations should be considered to atone for the damage.

“Algorithms leave imprints that are often immutable,” he said. “Just because they are made of software, it doesn’t mean there’s an undo button there.

“We need people to understand that algorithms have consequences that outlive their own existence, and if that doesn’t bring us into a more mindful, ethical way of thinking about deployments, I’d be very sad.”

The Algorithmic Imprint was co-authored with Ranjit Singh and Jacob Metcalf from Data & Society Research Institute and Professor Mark Riedl from the School of Interactive Computing.

Vanessa Oguamanam has researched the correlation of digital tools and how often Black women in perinatal stages use them to improve their mental health.

According to the Anxiety and Depression Association of America, Black women are at “higher risk for experiencing perinatal and postnatal anxiety disorders such as depression, anxiety, obsessive compulsive disorder, and posttraumatic stress disorder." The risk for PMADs is estimated to be double that of the general population.

[MICROSITE: Georgia Tech at CHI 2023]

The problem has worsened since the Covid-19 pandemic, says Oguamanam, a Ph.D. student in the School of Interactive Computing under the advisement of Associate Professor Andrea Parker, founder and director of the Wellness Technology Research Lab. Oguamanam has spent most of her doctoral career researching technology designed to benefit the health of Black women.

“I had friends who became new moms during the pandemic and just seeing the extra amount of stress that they were enduring in addition to balancing new childcare responsibilities led me to start thinking of potential ways I could address this mental health crisis with technology.” she said.

“Mental health is one of the leading complications during pregnancy and childbirth, and it’s a contributing factor to some maternal deaths. The pandemic exacerbated all of that. We’re seeing rates that are skyrocketing.”

In a paper that was accepted at the 2023 Conference on Human Factors in Computing Systems (CHI), Oguamanam and Parker surveyed 101 pregnant and postpartum Black women. They found 34% reported moderate to severe anxiety, while 41% percent expressed having moderate to severe psychological distress, and 74% experienced a high level of postnatal depressive symptoms.

Oguamanam and Parker also studied participant interaction with four main forms of technology — social media, apps, self-tracking devices, and video calls.

“There’s been very little work investigating how we can design digital mental health tools to support Black pregnant and postpartum women’s needs,” Parker said. “We’re trying to understand what their current use and satisfaction level is with existing platforms. We need this foundational understanding to drive future design efforts.”

The research indicates that income and education levels were significant variables among the women surveyed. Of the 101 participants surveyed, 49 identified as low income, and 43 percent identified as middle to upper income. Forty-three held less than a bachelor’s degree while 58 held a bachelor’s degree or higher.

Those with higher incomes and education tended to use apps and self-tracking devices more frequently. The use of video calls varied among pregnancy status and the area of the U.S. where participants lived. Women who were pregnant and lived in the South used video calls most frequently.

Social media was widely used among all demographics.

One of the main takeaways from the study is that participant feedback shows the “one-size-fits-all” approach that digital mental health interventionists often take in their design methods can be insufficient for meeting the needs of pregnant and postpartum Black women. Oguamanam said the societal problems of systemic racism and barriers to healthcare that Black women experience aren’t often considered in such efforts.

“Years and years of experiencing racial and gendered discrimination have impacted the stress levels of Black women and their overall well-being,” she said.

“It’s important to emphasize that when we’re thinking about health disparities among racial groups, there can be a tendency to think it just boils down to differences in socioeconomic status,” echoed Parker. “But many of these disparities persist when we compare higher income groups of black women to another racial ethnic group. These inequitable differences reflect a broader set of structural forces that create barriers to healthcare access and quality and increased exposure to mental health threats.”

Oguamanam and Parker also found that 97% of the women surveyed embraced the identity of the strong Black woman, a representation that has been explored at length by social science researchers.

Researchers agree the external factors of systemic racism and healthcare barriers tend to push Black women toward that identity.

“It’s a role that a number of Black women tend to identify with, either consciously or subconsciously,” Oguamanam said. “It’s the idea of presenting an image of strength and feeling like you have to take care of you, your family, and your community and that you’re responsible for carrying the weight of the world on your shoulders.”

Oguamanam’s and Parker’s study indicates women with greater adoption of the strong Black woman persona tended to use self-tracking devices with greater frequency. That trend could be attributable to those devices offering a sense of autonomy, Parker said.

“The whole vision of self-tracking devices is that you can take care of yourself,” Parker said. “You can monitor your own well-being and oversee collecting data and managing your own health. That type of platform might be more appealing to individuals who have a resistance toward being vulnerable.”

These findings only scratch the surface, and Oguamanam and Parker hope to shift current methods and discussions surrounding digital mental health toward a more inclusive environment that includes the experience of pregnant and postpartum Black women.

“More research is needed to investigate these hypotheses, and ultimately design and demonstrate the effectiveness of digital tools that support the wellbeing of pregnant and postpartum Black women,” Parker said. “Such innovations can help us make necessary strides toward achieving maternal mental health equity.”

But a team of graduate students within the School of Interactive Computing have explored how technology can help circumvent these barriers so that women can engage in freer communication on stigmatized health issues.

Hannah Tam, Karthik Bhat, and Priyanka Mohindra conducted research to study how safe spaces could be curated to support 35 women of Indian origin in discussing subjects that are otherwise considered taboo.

Neha Kumar, an associate professor who teaches jointly with the School of Interactive Computing and the Sam Nunn School of International Affairs, served as advisor to the students. Kumar is the director of the Tandem Lab, which works to explore cultural taboos and investigate their impact on health and well-being among women and gender minorities internationally.

[MICROSITE: Georgia Tech at CHI 2023]

During the two-week study, the researchers provided the group with discussion prompts on stigmatized topics such as menstrual health, sexual wellbeing, fitness, body image, diet and exercise, and mental health.

“Taboos persist because Indian culture is still largely patriarchal,” Bhat said. “The cultural norm is that you don’t talk about these things with other people, you don’t talk about them with other genders, you don’t talk about these things outside the home. It becomes hard to seek care where it’s necessary and community where it’s necessary.”

The restrictions of societal taboos escalate when adolescent girls begins their menstrual cycles, Mohindra said.

“Everything about how they’re treated changes,” she said. “They’re not able to go to temples because it’s considered impure. They’re not allowed to serve food or go into the kitchen. Nobody should know you’re going through that, even though it’s a natural process.”

“Anything that has to do with intimacy or anything sexual and a woman is involved, that’s looked down at no matter what age you are,” she said. “Girls are afraid to ask their mothers about these things.”

Tam said she and her colleagues used legitimate peripheral participation (LLP) as a framework to analyze the social learning that took place in the Whats App group they established for the study. The LLP framework creates social interactions among community members that enable them to share knowledge and experiences while allowing other members to observe and learn without actively participating.

“We found that learning happened at all levels, which was applicable to both core members and peripheral members, or members who might not have been as active in the group,” Tam said.

The group of women participated with partial anonymity, which also provided a sense of security and comfort to most users. Using WhatsApp only required members to provide their phone numbers, which helped them conceal their identities to minimize the risk of facing judgment in their communities.

The group conducted follow-up interviews with 10 of the 35 participants and found that some had started connecting with other group members offline, which Tam said was a step forward from where they had started.

“People found relief in being able to talk openly,” Tam said. “One member said she hadn’t felt comfortable saying the word ‘period’ aloud — even among her close friends.

“As members were unpacking their experiences with taboos outside the group, that led to a lot of members questioning traditional systems and social structures.”

The goal of the project, Bhat said, was not about making a direct impact on the health of the women participating. Instead, the researchers aimed to find ways in which social media platforms could enable sharing of sensitive information and equip participants with the ability to navigate cultural barriers.

“Our findings suggested that this space and our intervention gave them the avenue to learn how to engage on stigmatized health topics and then take these conversations out into the world,” he said. 

“There needs to be dedicated efforts to educating at various levels so that people who are in positions of power, such as governments and healthcare authorities, recognize and address the challenges in health communication that we saw, and work to address them. This is critical for women’s health and wellbeing worldwide. But given that these problems exist and are not going away anytime soon, how can technology support us in addressing this gap now?”