{"662312":{"#nid":"662312","#data":{"type":"news","title":"Research Paves Way for Home Robot that Can Tidy a House on Its Own","body":[{"value":"\u003Cp\u003EStruggling with keeping your home clean and organized? You may soon have an extra set of hands to help around the house.\u003C\/p\u003E\r\n\r\n\u003Cp\u003EImagine a home robot that can keep a house tidy without being given any commands from its owner. Well, the next step in home robotics is here \u0026mdash; at least virtually.\u003C\/p\u003E\r\n\r\n\u003Cp\u003EA group of doctoral and master\u0026rsquo;s students from Georgia Tech\u0026#39;s School of Interactive Computing, in collaboration with researchers from the University of Toronto, believe they have created the benchmark for a home robot that can keep an entire house tidy.\u003C\/p\u003E\r\n\r\n\u003Cp\u003EIn their paper,\u0026nbsp;\u003Cem\u003EHousekeep: Tidying Virtual Households Using Commonsense Reasoning\u003C\/em\u003E, Georgia Tech doctoral candidates \u003Cstrong\u003EHarsh\u003C\/strong\u003E \u003Cstrong\u003EAgrawal\u003C\/strong\u003E and \u003Cstrong\u003EAndrew\u003C\/strong\u003E \u003Cstrong\u003ESzot\u003C\/strong\u003E, master\u0026rsquo;s students \u003Cstrong\u003EArun\u003C\/strong\u003E \u003Cstrong\u003ERamachandran\u003C\/strong\u003E and \u003Cstrong\u003ESriram\u003C\/strong\u003E \u003Cstrong\u003EYenamandra\u003C\/strong\u003E, and \u003Cstrong\u003EYash\u003C\/strong\u003E \u003Cstrong\u003EKant\u003C\/strong\u003E, a former research visitor at Georgia Tech who is now a doctoral candidate at Toronto, set out to prove an embodied artificial intelligence (AI) could conduct simple housekeeping tasks without explicit instructions.\u003C\/p\u003E\r\n\r\n\u003Cp\u003EUsing advanced natural language processing machine learning techniques, the students have successfully simulated the robot exploring a virtual household, identifying misplaced items, and putting them in their correct place.\u003C\/p\u003E\r\n\r\n\u003Cp\u003EKant said most robots in embodied AI are given specific instructions for different functions, but the students wanted to be sure the robot could achieve task completion without instructions in simulation before moving on to real-world testing.\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026ldquo;In the actual world, things are difficult,\u0026rdquo; Kant said. \u0026ldquo;Training robots in the real world \u0026mdash; they move around slowly; they will bump into things and people. So, we do it in simulation because you can run things at a faster speed, and you can have multiple virtual robots running.\u0026rdquo;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cstrong\u003EDhruv\u003C\/strong\u003E \u003Cstrong\u003EBatra\u003C\/strong\u003E, an associate professor in the School of Interactive Computing and a research scientist with Meta AI, and \u003Cstrong\u003EIgor\u003C\/strong\u003E \u003Cstrong\u003EGilitschenski\u003C\/strong\u003E, an assistant professor of mathematical and computational sciences at Toronto, served as advisors on the paper, which was accepted to the 2022 European Conference on Computer Vision, Oct. 23-27 in Tel Aviv, Israel.\u003C\/p\u003E\r\n\r\n\u003Ch4\u003E\u003Ca href=\u0022https:\/\/sites.gatech.edu\/ml-eccv-2022\/\u0022\u003E[FULL COVERAGE: Georgia Tech at ECCV 2022]\u003C\/a\u003E\u003C\/h4\u003E\r\n\r\n\u003Cp\u003EIn the virtual simulation, the robot spawned in a random section of the house and immediately began looking for misplaced objects. It correctly identified a misplaced lunchbox in a kid\u0026rsquo;s bedroom and moved it to the kitchen. It also located some toys left in the bathroom and moved them to the kid\u0026rsquo;s bedroom.\u003C\/p\u003E\r\n\r\n\u003Cp\u003EAgrawal said the goal of the project from the beginning was to have the robot mimic commonsense reasoning that any human would have in tidying a house. Through surveys, the team collected rearrangement preferences for 1,799 objects in 585 placements in 105 rooms.\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026ldquo;We collected human preferences data,\u0026rdquo; Agrawal said. \u0026ldquo;We asked people where they like to keep certain objects, and we wanted robots to have a similar notion of cleanliness in a tidy home.\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026ldquo;You don\u0026rsquo;t provide instructions when you ask the kids to clean up the house. It\u0026rsquo;s commonsense. You know certain things go in certain places. You know Lego blocks don\u0026rsquo;t belong in the bathroom. We thought it\u0026rsquo;d be cool if it could clean up the house without specifying instructions. As humans, we can do a bunch of these tasks without being given specific instructions.\u0026rdquo;\u003C\/p\u003E\r\n\r\n\u003Cp\u003ECreating the simulation had several challenges. These included getting the robot to use reason about the correct placement of new objects, getting the robot to adapt to new environments, and getting it to work through choices when there are multiple correct locations a misplaced object could go.\u003C\/p\u003E\r\n\r\n\u003Cp\u003ESzot said what attracted him to the project was the idea of creating a robot that didn\u0026rsquo;t need to be told where to put something, whereas in his previous work, that\u0026rsquo;s exactly what he had to do.\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026ldquo;If you wanted it do something like clean up the house, you would have to tell it, \u0026lsquo;Hey, robot, move that object to there,\u0026rsquo;\u0026rdquo; Szot said. \u0026ldquo;It\u0026rsquo;s very tedious to specify that. We took the first step of saying let\u0026rsquo;s give the robot some commonsense reasoning. It might not be specific to a person; it might just be capturing more generally what people think, but it captures a lot of important situations. It\u0026rsquo;s able to handle most of those situations in which people agree the object belongs there or the object doesn\u0026rsquo;t belong there.\u0026rdquo;\u003C\/p\u003E\r\n\r\n\u003Cp\u003EUsing text from the internet, the team informed the AI that drives the robot by fine-tuning a large language model based on human preferences.\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026ldquo;The way we approached solving this problem is we took this external source of knowledge from text on the internet and these language tasks, and so from natural language processing we took that information and used it to give our robot some idea of this common sense,\u0026rdquo; Szot said. \u0026ldquo;It wasn\u0026rsquo;t purely from the house it learned how to do these things. From articles or texts online, it was able to distill this commonsense reasoning ability and then apply it.\u0026rdquo;\u003C\/p\u003E\r\n\r\n\u003Cp\u003EKant said using language models allows the AI to distinguish between objects and whether those objects should go together. He added that he thinks that the language model used to train the AI can be fine-tuned by extracting content from web articles related to housekeeping.\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026ldquo;Language models have shown very promising results in trying to extract semantics, like whether two things \u0026mdash; say an apple and fruit basket \u0026mdash; go together in a household,\u0026rdquo; Kant said.\u003C\/p\u003E\r\n\r\n\u003Cp\u003EThe team is just at the tip of the iceberg, and the virtual simulation serves only as a proof of concept. It\u0026rsquo;s a long-term project that will continue to explore new possibilities, which include creating a robot that can tidy a household according to specific user preferences.\u003C\/p\u003E\r\n\r\n\u003Cp\u003EBut the successful use of NLP methods to inform a novel AI could break new barriers in the creation of new systems in which organization is the focus.\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026ldquo;It\u0026rsquo;s a benchmark for the rest of the community to use,\u0026rdquo; Szot said. \u0026ldquo;Hopefully this is something for people to gather behind to focus on this very realistic task setting of cleaning the house. We showed that you can create these embodied agents that can use this external knowledge and learn commonsense and use it in embodied robotic settings.\u0026rdquo;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026ldquo;I think the data that we collected is pretty significant in the sense that we now have a few hundred annotations for where each object should go in houses and where they\u0026rsquo;re likely to be found in untidy houses, and I think that information can guide a lot of systems,\u0026rdquo; Agrawal added. \u0026ldquo;I feel like we are starting to now see people saying all these annotations can be used for building their own systems and benchmarks.\u0026rdquo;\u003C\/p\u003E\r\n","summary":null,"format":"limited_html"}],"field_subtitle":"","field_summary":"","field_summary_sentence":[{"value":"A group of doctoral and master\u2019s students from Georgia Tech\u0027s School of Interactive Computing believe they have created the benchmark for a home robot that can keep an entire house tidy."}],"uid":"32045","created_gmt":"2022-10-19 15:09:13","changed_gmt":"2022-10-19 20:02:41","author":"Ben Snedeker","boilerplate_text":"","field_publication":"","field_article_url":"","dateline":{"date":"2022-10-19T00:00:00-04:00","iso_date":"2022-10-19T00:00:00-04:00","tz":"America\/New_York"},"extras":[],"hg_media":{"662342":{"id":"662342","type":"image","title":"Housekeep","body":null,"created":"1666204859","gmt_created":"2022-10-19 18:40:59","changed":"1666204859","gmt_changed":"2022-10-19 18:40:59","alt":"Housekeep is a benchmark to evaluate commonsense reasoning in the home for embodied AI. I","file":{"fid":"250840","name":"housekeeping-algorithm.jpeg","image_path":"\/sites\/default\/files\/images\/housekeeping-algorithm.jpeg","image_full_path":"http:\/\/hg.gatech.edu\/\/sites\/default\/files\/images\/housekeeping-algorithm.jpeg","mime":"image\/jpeg","size":88451,"path_740":"http:\/\/hg.gatech.edu\/sites\/default\/files\/styles\/740xx_scale\/public\/images\/housekeeping-algorithm.jpeg?itok=N8VKYqsO"}},"662343":{"id":"662343","type":"image","title":"Housekeep research team collage","body":null,"created":"1666204942","gmt_created":"2022-10-19 18:42:22","changed":"1666204942","gmt_changed":"2022-10-19 18:42:22","alt":"Housekeep research team collage","file":{"fid":"250841","name":"authors_housekeeping-bot-copy_v2.2.jpg","image_path":"\/sites\/default\/files\/images\/authors_housekeeping-bot-copy_v2.2.jpg","image_full_path":"http:\/\/hg.gatech.edu\/\/sites\/default\/files\/images\/authors_housekeeping-bot-copy_v2.2.jpg","mime":"image\/jpeg","size":227143,"path_740":"http:\/\/hg.gatech.edu\/sites\/default\/files\/styles\/740xx_scale\/public\/images\/authors_housekeeping-bot-copy_v2.2.jpg?itok=6pxpNCGZ"}}},"media_ids":["662342","662343"],"related_links":[{"url":"https:\/\/sites.gatech.edu\/ml-eccv-2022\/","title":"Georgia Tech at ECCV 2022"}],"groups":[{"id":"576481","name":"ML@GT"},{"id":"66442","name":"MS HCI"},{"id":"50876","name":"School of Interactive Computing"},{"id":"434391","name":"ECE M.S. Thesis Defenses"},{"id":"434381","name":"ECE Ph.D. Dissertation Defenses"},{"id":"434371","name":"ECE Ph.D. Proposal Oral Exams"},{"id":"1188","name":"Research Horizons"}],"categories":[],"keywords":[{"id":"1356","name":"robot"},{"id":"191487","name":"eccv"},{"id":"191488","name":"tidy"},{"id":"2483","name":"interactive computing"},{"id":"187915","name":"go-researchnews"}],"core_research_areas":[{"id":"39521","name":"Robotics"}],"news_room_topics":[],"event_categories":[],"invited_audience":[],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[{"value":"\u003Cp\u003ENathan Deen, Communications Officer\u003C\/p\u003E\r\n","format":"limited_html"}],"email":["ndeen6@gatech.edu"],"slides":[],"orientation":[],"userdata":""}}}