It began with a focus on helping computer systems incorporate human language into their perception of the world. Could a robot tell humans that a heavy object is straining its arm? Does it know that a container makes a sound when it is picked up or dropped?
Now, Thomason’s research has become more human-centered, which is one reason he feels at home as he joins the faculty at Georgia Tech’s School of Interactive Computing as an associate professor.
Thomason comes to Georgia Tech after spending five years as an assistant professor at the University of Southern California’s Thomas Lord Department of Computer Science.
He will move his Grounding Language in Actions, Multimodal Observation, and Robots (GLAMOR) lab to Tech. The lab explores methods for connecting natural language processing to robotics.
While pursuing his computer science Ph.D. at the University of Texas, Thomason researched early NLP systems before large-language models such as ChatGPT and Google Gemini were introduced.
Thomason then expanded into robotics to explore how roboticists can create new ways to communicate with robots using computational representations of language.
“The idea was you could do something that looked like robot programming without as much engineering work if you used one of these coding modules to create a language-code interface,” Thomason said.
When OpenAI and Google released their commercial LLMs in 2022, Thomason felt a seismic shift.
“That area of NLP and robotics became more crowded and accessible with the advent of (LLMs),” Thomason said. “It created a huge inroad to roboticists being able to plug and play language technologies in a way they couldn’t before. Robotics hardware has also become much cheaper and more standardized.”
But as LLMs became universal tools accessible to everyone, Thomason said he started thinking more about how he could improve optimization for users.
“I became most interested in how language interfaces, starting in 2022, were suddenly everywhere,” he said. “Everybody had a chatbot on their phone. I wanted to see what we could do as researchers to make the interaction between an AI system like a language model and a person smoother and more beneficial.”
Thomason said he felt he needed a new environment where end users were at the center of the technology design process. He was drawn to the School of Interactive Computing for its reputation in human-computer interaction research.
“That was a big factor in my decision to move to the School of Interactive Computing, where you value that kind of human-centered computing,” he said.
“With the strong HCI researchers here, I have an opportunity to learn more from my colleagues about how to run high-quality human-subject studies, conduct participatory design, and identify actual users of a potential system and talk to them about what would be helpful.”
]]>Karamcheti said there are times when he doesn’t see any humans at all. Those instances reinforce the perception that robots are replacing humans rather than assisting them.
“You have humanoid robots doing things like folding laundry, and you see videos of it on social media,” Karamcheti said. “It bothers me that in all these videos, I see the robot doing these tasks alone without a single person around them. True human-robot collaboration is missing.”
As he joins the Georgia Tech School of Interactive Computing as an assistant professor, Karamcheti seeks to improve human-robot collaboration and maximize output through his research.
Karamcheti comes to Tech after earning his Ph.D. in computer science (CS) from Stanford University. He also holds bachelor’s degrees in CS and literary arts from Brown University.
In addition to human-robot interaction, he has conducted research on fine-tuning vision-language-action (VLA) models.
VLAs are artificial intelligence (AI) systems typically built by fine-tuning vision-language models (VLMs) or large language models (LLMs). They use visual perception, natural language, and physical action to perform tasks.
“I have a high-level vision of what I think robotics should be, but there are many winding paths I like to take along the way as I try to understand why something works the way it does,” Karamcheti said.
Karamcheti said his vision was inspired by the movie Iron Man and by how the main character, Tony Stark, interacts with robots in his lab.
“I don’t want a robot that replaces all the work I’m doing in my life,” he said. “I want a robot that’s embedded in my life like my laptop or smartphone.
“They’re making it possible for me to do more things. I also want to make it so that robots start doing more and learning more when I leave the room.”
Karamcheti said the School of IC is a “dream” job for him because of its renowned faculty in human-robot interaction. It’s the ideal place to keep humans at the center of his research.
“We must design models around interaction and the idea that there’s going to be a person in the room. Robots don’t need to be perfect at any one thing. They just need to be able to ask for help when it’s needed and learn from those teaching interactions.”
Karamecheti said many robotics researchers are pursuing robots that are as close to perfect as possible so they can operate without human oversight. However, the amount of money and data needed to get there is substantial.
“I think you see diminishing returns with scale. If a robot performs tasks correctly 90% of the time, the data required to go from 90% to 95% is much less than that required to go from 95% to 99%. Many robotics companies and labs are investing to reach a 99% success rate,” he said.
Karamcheti added that the pursuit of perfection shouldn’t prevent a robot from being deployed. They could be deployed at 95% and increase their success rate by working with and learning from humans.
“Once I have a robot that’s 95% capable, I understand that 5% of failures will occur for a specific task. If I know a human is always going to be near the robot, I can design a system that sequences tasks and anticipates human intervention.”
Karamcheti is recruiting students to join his lab who bring new perspectives to robotics.
“I’m looking for students who aren’t afraid to challenge common assumptions and who enjoy hacking and building,” he said. “I want to work with students who have strong opinions about how robotics should look.”
]]>Karamcheti comes to Tech after earning his Ph.D. in computer science (CS) from Stanford University. He also holds bachelor’s degrees in CS and literary arts from Brown University.
In addition to human-robot interaction, he has conducted research on fine-tuning vision-language-action (VLA) models.
]]>Dunkin is an IISE Fellow, the highest classification of IISE membership that recognizes outstanding leaders of the profession who have made significant, nationally recognized contributions to industrial and systems engineering. She has served as chief information officer at the U.S. Department of Energy and the U.S. Environmental Protection Agency, as well as chief strategy and innovation officer at Dell Technologies. She has also held numerous leadership positions at Hewlett Packard, largely in the areas of manufacturing, research and development and IT. She currently serves as a distinguished external fellow with Georgia Tech’s Strategic Energy Institute.
Dunkin holds appointments as a distinguished professor of the practice in Georgia Tech’s School of Cybersecurity and Privacy and in the Jimmy and Rosalynn Carter School of Public Policy. She is also a distinguished external fellow at the Georgia Tech Research Institute (GTRI).
“The Emerging Technologies forum is a great opportunity for practicing ISEs to get exposure to a range of emerging technologies in a short period of time and identify those that they want to experiment with and implement in their organizations,” Dunkin said.
The Aug. 26 event, which will be held virtually from 1-5 p.m. EDT, will explore technologies shaping the future of industrial and systems engineering, including artificial intelligence, digital twins, and robotics and autonomy.
As event chair, Dunkin will be joined by experts from the healthcare, aerospace, logistics, manufacturing and technology industries, including representatives from Royal Caribbean, The Boeing Company, UPS and other organizations.
The complimentary event is open to engineers and improvement professionals of all backgrounds. Registration is available through the IISE Emerging Technologies Forum website.
For most, that heart-pounding uncertainty ends the moment the foot finds solid ground. But for many individuals living with conditions like stroke or spinal cord injury (SCI), that sense of disconnect is a permanent reality.
“These conditions of course have a huge effect on our ability to move around and be independent — but the other side of it is the sensory feedback that we lose,” says Matthew Flavin, an assistant professor in the School of Electrical and Computer Engineering. Most rehabilitation treatments primarily focus on restoring movement, but “even if you have motor control, if you can’t feel when your foot's touching the ground it can be really hard for you to move around safely.”
In a new study published in Proceedings of the National Academy of Sciences, Flavin and an interdisciplinary team of researchers introduce a way to bridge this gap: a wearable “sensory substitution” system that translates foot pressure into high-tech patterns of heat and vibration they can feel elsewhere.
The system uses high-resolution pressure-sensing insoles designed by the team, which are placed inside a user's shoes to record how their weight shifts in real-time. This data is streamed via Bluetooth to a flexible, skin-conformable array of haptic receivers worn on the forearms, a part of the body that often retains sensation in SCI. The receivers give quick pressure feedback through vibration, while also alerting the user to longer-term pressure “hotspots” through heat.
“One of the limitations of a lot of approaches in haptics is that you're having to map a missing sense onto a completely different sense,” says Flavin. “We’re keeping the type of information that we're missing, which is the distribution of pressure, and we're just basically putting it on a different part of their body.”
Rerouting the lost sensation was key to making the device intuitive to learn. Participants were able to correctly identify the “feel” of the ground through their arms with high accuracy within a mere two-hour session. When tested with a small group of participants with stroke or SCI, the wearable significantly improved standing balance and led to steadier walking.
“What’s encouraging about these early results is that participants appeared to use the feedback in ways that supported balance and walking,” says John Rogers, a materials science and engineering professor at Northwestern University who collaborated on this study. “Our study suggests that providing pressure information through another part of the body could be a practical path for helping people compensate for lost sensation.”
While vibration provides immediate feedback for walking and balance, the team views the thermal feedback as a tool for long-term health. Heat is a slower, low-frequency signal that could alert patients to pressure hotspots, potentially preventing diabetic foot ulcers or pressure injuries for those who are bedridden or use wheelchairs.
The small, lightweight system is completely untethered, making it suitable for use during daily activities in and outside the clinic. It’s also highly adaptable to different injury types, which is ideal for conditions as variable as stroke, SCI, and diabetic neuropathy. Placement of the haptic receivers can be adjusted based on where a patient has the most sensation, and the sensitivity of the insoles can be tailored to each patient.
As a member of several of Georgia Tech’s Interdisciplinary Research Institutes — the Institute for Neuroscience, Neurotechnology, and Society, the Institute for Robotics and Intelligent Machines, and the Parker H. Petit Institute for Bioengineering and Biosciences — Flavin credits the project’s success to an interdisciplinary effort and deep engagement with clinicians and patients.
“This reinforces the importance of really engaging with your stakeholders very early on,” says Flavin. “If you're not continually refining that concept with those stakeholders, you quickly find that they might be looking for something that your device isn't delivering.”
With new funding from the National Science Foundation (NSF), the team is now working to make the technology even smaller and more reconfigurable, moving closer to a standard wearable for daily clinical use.
DOI: https://doi.org/10.1073/pnas.2536577123
]]>Photos:
Maxwell Guberman
RAI calls the bicycle robot an ultra-mobility vehicle (UMV). It can reach a height of 3 feet and can jump from the floor onto a platform.
The contributions of a Georgia Tech Ph.D. student helped make these feats possible through a robot control policy he developed.
Jeonghwan Kim, who is pursuing a Ph.D. in robotics under the advisement of Associate Professor Sehoon Ha, spent two semesters interning at RAI. His task was to design a policy to teach the UMV to land after a flip.
The result was iterative motion imitation (IMI), a novel method that imitates flip trajectories generated from prior examples. Kim said the robot bases its flip on a demonstration, and human engineers reconstruct and refine the flip path through simulation to fill in the gaps.
“To guide the robot to flip, we started with an imperfect trajectory generated by a motor-based controller and then ran simulations,” Kim said. “It’s an unstable trajectory, but we use it as a guide to train a single policy that can track it as it lands and tries to balance itself.”
Kim interned under the supervision of Shamel Fahmi, a research scientist at the RAI Institute. RAI has been developing the UMV for nearly three years.
“We wanted to work on a different robot morphology that wasn’t legs or legs with wheels,” Fahmi said. “That’s when we thought of working with bikes.
“We want to merge the athleticism of (Boston Dynamics’) Atlas with the mobility of a bike. We wanted a robot that can go anywhere, do parkour, and acrobatics.”
Fahmi said that before Kim arrived, the research team had trouble getting the UMV to land consistently without breaking or falling.
The UMV has two joints — an upper and a lower. The upper joint contains the motors and pulls the lower joint along as it propels into the air. The problem is getting the lighter lower joint to absorb the impact of landing without being crushed by the heavier upper joint.
“That’s what brings reinforcement learning into the equation,” Fahmi said. “We teach the robot to minimize its impact on the ground to land gracefully.”
Fahmi said that Kim proved the imitation examples the robot learns from don’t have to be perfect. The process takes some time, but all it needs is a rough idea to get started.
“You can have an imperfect sketch and then constantly refine it,” Fahmi said. “The first time, it’s not going to go well.
“We don’t care about torque or power limits as long as it does the motion. Then we’ll have a slightly better reference, repeat it, and imitate it again. In every iteration, we can add more parameters.”
Kim said he felt the pressure of time constraints during his two semesters with RAI as he worked to achieve consistent, successful landings. Even though he had multiple UMVs to experiment with, they broke down dozens of times. Each time one broke, a hardware team at RAI had to repair it.
“There was a lot of pressure to not only get this working before my internship ended, but also knowing there are costs behind every failed attempt, and every time the robot breaks, it takes time to repair it,” Kim said.
“It took almost five months for it to land without breaking. Then we needed two more months for it to stay balanced after the landing. It requires a lot of engineering effort to achieve a robust control policy for a safe flip.”
By the time Kim left RAI, the IMI policy had achieved consistent, seamless landings.
“The jump right now is what we call the visitor demo,” Fahmi said. “If there are guests coming over to see it, we want to show them something that is extremely impressive, but also, more importantly, extremely reliable. It never fails.
“It was only possible because of the huge effort we put into designing, maintaining, and continuously improving the robot.”
Kim authored a paper on his framework and will present it at this week’s International Conference on Robotics and Automation (ICRA) in Vienna.
For more information about the UMV project, please visit the RAI blog or watch their video on YouTube.
]]>Jeonghwan Kim, who is pursuing a Ph.D. in robotics under the advisement of Associate Professor Sehoon Ha, spent two semesters interning at RAI. His task was to design a policy to teach the UMV to land after a flip.
]]>The new technology is also set to revolutionize the scale of policy training data collection, which is essential to advancing robotic capabilities and meeting growing production demand.
COBALT is a mobile app that turns smartphones into controllers for robot arms. With a secure Wi-Fi connection to a server, users can move their phones in any direction, and the robot arm will mirror the motion — from anywhere in the world.
Ayush Agarwal, a Ph.D. student in Georgia Tech’s School of Interactive Computing who leads a research team developing COBALT, said it works like the games people play on smartphones. Users can press a button to have the arm grasp an object, move it, and release it with another button.
Agarwal conducted several user studies with participants in nine countries who remotely operated robot arms inside Georgia Tech’s People, AI & Robotics (PAIR) Lab. The lab is directed by Assistant Professor Animesh Garg, who advises Agarwal.
“We built an entire distribution system for remote teleoperation scaled to where we had people from Indonesia, India, and Pakistan operating for us,” Agarwal said. “They were novice operators who had never done it before. By collecting data from these new users, we showed that we can train policies to automate certain tasks.”
Garg envisions a world where data collection for policy training is done through crowdsourcing. He began working toward this goal 10 years ago as a postdoc at Stanford University, when he developed RoboTurk, an earlier version of COBALT.
“There is a large-scale data collection requirement for mass robot production to be possible, and it will not be solved purely through simulation,” Garg said.
“Our idea was, what if we could get almost every person on the planet to be a passive source for data collection? There are almost five billion people who have smartphones and know how to use them.”
Another major implication of COBALT could be expanded access to CS and robotics education.
Students can learn to operate a robot remotely in any classroom. In fact, Garg and his lab recently hosted students from Midtown High School in Atlanta to demonstrate COBALT and let them control robot arms from a phone.
Garg also sees the possibility of a “gig economy” in which people pay remote operators to control assistive robots in their homes and complete household chores for them.
“It could be Uber for robots,” he said. “People who want to log onto the platform can do so at their convenience and for as long as they want.”
Companies with robot-dependent labor tasks could also use the platform to enable human oversight.
“If I deploy a robot in a factory that achieves high autonomy for most tasks, but there are still times it needs help, a human could operate the robot from anywhere in the world,” Garg said.
Agarwal’s studies showed that people prefer to interact with and control a robot using a smartphone rather than virtual reality (VR) headsets, controllers, keyboards, mice, or other devices.
“The phone is a more intuitive interface and can provide data quality that’s on par with other commonly used devices,” he said.
Agarwal also said there is minimal latency in the video feed sent back to operators on the other side of the world. That’s because the amount of data being processed is small.
The data is carried over Web Real-Time Communication (WebRTC), the same technology used by many streaming services and web conferencing platforms such as Zoom and Google Meet.
“There’s a connection from your phone to the teleoperation server, which is connected to the robots,” Agarwal said.
“Then there’s another connection from the teleoperation server back to the user, which allows for a video stream. We need low latency on both because you don’t want the user to move their phone and wait 10 seconds to see the visual feed.”
Agarwal is the co-lead author of a paper on COBALT that is being presented at the IEEE International Conference on Robotics and Automation this week in Vienna. He said the paper stands out because it has moved from theory to the implementation of an entire distribution network.
“The real novelty of our paper is the systems that we build around it to actually support the scaling of remote operation and data collection at a global level,” he said.
]]>