{"691044":{"#nid":"691044","#data":{"type":"news","title":"Researchers Use GeoGuessr Champion to Test Geolocation Accuracy in VLMs","body":[{"value":"\u003Cp\u003EThe house in the distance, with a red, hip-shaped roof and white walls, tells Radu Casapu that this place is probably somewhere in Spain or Portugal.\u0026nbsp;\u003C\/p\u003E\u003Cp\u003EThe surrounding trees resemble those of a eucalyptus forest, which could indicate northern Portugal or the Spanish region of Galicia.\u003C\/p\u003E\u003Cp\u003EIt\u2019s the signposts on the road that give it away. They are flat and wide, which is common in Spain but not in Portugal.\u003C\/p\u003E\u003Cp\u003ECasapu, a master\u2019s student in Georgia Tech\u2019s School of City and Regional Planning, correctly reasons that the picture of a road he\u2019s looking at is in Galicia.\u003C\/p\u003E\u003Cp\u003EGive Casapu a photo, and he will likely be able to tell you where it was taken.\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u201cI start with infrastructure clues that are specific to a country, region, state or province,\u201d Casapu said. \u201cThey include roads or electricity poles, which often remain consistent throughout a country. Once you narrow down the country, you can use more specific factors like vegetation, specific landscapes, or architecture, because these are very nuanced. It\u2019s a top-down approach.\u201d\u003C\/p\u003E\u003Cp\u003EThis is why Casapu is the reigning\u0026nbsp;\u003Ca href=\u0022https:\/\/news.gatech.edu\/news\/2025\/09\/23\/georgia-tech-graduate-student-wins-geoguessr-world-championship\u0022\u003EGeoGussr World Champion\u003C\/a\u003E \u2014 and the ideal expert to test vision-language models (VLMs) on how good they are at geolocation.\u003C\/p\u003E\u003Cp\u003EGeoGuessr is a geography browser game launched in 2013 that invites players to guess the location of random Google Street View images. Casapu was already known as one of the top players in the world before he won the third annual GeoGussr World Championship in September.\u003C\/p\u003E\u003Cp\u003EAt the beginning of the spring 2025 semester, School of Interactive Computing professor James Hays reached out to Casapu and invited him to collaborate on a new project. Hays was looking to create a dataset to evaluate VLMs\u0027 geolocation ability and reasoning.\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u201cVLMs are surprisingly good at geolocation right out of the box, even when they\u2019re not trained to be good at it,\u201d Hays said.\u003C\/p\u003E\u003Cp\u003EHays and his colleagues, associate professors Alan Ritter and Wei Xu, took issue with many AI companies claiming that the VLMs they were releasing were not good at geolocation.\u003C\/p\u003E\u003Cp\u003E\u201cWhen Open AI released GPT 4 Vision, there were privacy concerns about the model\u2019s ability to geolocate someone based on photos they\u2019ve shared on the internet,\u201d Ritter said. \u201cOpen AI said this wasn\u2019t a concern and claimed the model wasn\u2019t good at geolocation beyond being able to recognize a city or famous monument. We found that wasn\u2019t the case. These VLMs are state-of-the-art at image geolocation tasks.\u201d\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Ch4\u003E\u003Cstrong\u003EShow Your Work\u003C\/strong\u003E\u003C\/h4\u003E\u003Cp\u003EHays and Ritter enlisted a team of some of the world\u2019s top geolocators. It consisted of Casapu, Joshua Diao, a master\u2019s student in computer science, and Tejas Santanam, a Ph.D. student in industrial engineering. They each received 500 images to geolocate.\u003C\/p\u003E\u003Cp\u003ETeam members recorded their reasons for each of their answers. The result was GeoRC, the first benchmark for VLM geolocation performance, consisting of 800 \u201cground truth\u201d reasoning chains.\u0026nbsp;\u003C\/p\u003E\u003Cp\u003EHays and Ritter gave the same images to GPT 5, Gemini, Llama, and Qwen. The highest-performing model geolocated with 90% accuracy \u2014 not far off from the team\u2019s 96% score.\u003C\/p\u003E\u003Cp\u003EHowever, a major distinction showed up in the reasoning chains. While Casapu and Diao provided clear explanations for how they deduce each location, the VLMs either couldn\u2019t provide reasoning for their guesses or were vague in their answers.\u003C\/p\u003E\u003Cp\u003E\u201cThe research community has been demanding explanations from these models,\u201d Hays said. \u201cFor example, how do they know the location is in Italy?\u201d\u003C\/p\u003E\u003Cp\u003EHays has been researching this subject for almost 20 years. As a Ph.D. student at Carnegie Mellon University in 2008, he was the first researcher to take a machine learning approach to geolocation. He introduced a new algorithm that could estimate a geographic location from a single image.\u003C\/p\u003E\u003Cp\u003E\u201cWhen experts have audited these reasoning chains, we\u2019ve noted many suspicious or hallucinated attributes,\u201d he said. \u201cWhen they hallucinate a geographic property, why is it so often consistent with the correct guess?\u003C\/p\u003E\u003Cp\u003E\u201cI believe they\u2019re not revealing the true reasoning pathway that they used to determine the image was Italy. They\u2019re just implicitly recognizing that it was Italy for many reasons, then hunting for evidence to support that. Some of the things they say are true and supported by the image, and some are fabrications.\u201d\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\u003Ch4\u003E\u003Cstrong\u003EPractice Partner\u003C\/strong\u003E\u003C\/h4\u003E\u003Cp\u003ECasapu said there may be only a handful of GeoGuessr players who can currently outperform some top-tier VLMs in geolocation, and it may not be long before no one can.\u003C\/p\u003E\u003Cp\u003E\u201cI think it could be more difficult playing against these models than playing against another human because a human has the possibility of making mistakes at the top level,\u201d Casapu said. \u201cIf a well-trained model has that level of consistency, that is far beyond a normal person, and it would be much more difficult to beat.\u201d\u003C\/p\u003E\u003Cp\u003EHe added that working with Hays and competing against a machine improved his skill level and provided valuable practice ahead of the world championship.\u0026nbsp;\u003C\/p\u003E\u003Cp\u003E\u201cIt helps to take a step back and see why you\u2019re making the guesses that you are,\u201d he said. \u201cSince then, I\u2019ve taken a more methodical approach to how I practice. Writing these things down is a great way to see what you know and see why you make the guesses that you do. It\u2019s been a great training tool.\u201d\u003C\/p\u003E\u003Cp\u003ECasapu will defend his title at the 2026 GeoGussr World Championship in September.\u003C\/p\u003E\u003Cp\u003EHays, Ritter, Xu, Casapu, Diao, and Santanm are all co-authors of a paper on GeoRC along with lead author Mohit Talrej and Ph.D. students Ethan Mendes and Jim Thannikary. The paper will be presented next week at the 64th Annual Meeting of the Association for Computational Linguistics (ACL) in San Diego.\u003C\/p\u003E\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E","summary":"","format":"limited_html"}],"field_subtitle":"","field_summary":[{"value":"\u003Cp\u003EGeorgia Tech researchers James Hays, Alan Ritter, and Wei Xu partnered with reigning GeoGuessr World Champion Radu Casapu and other top players to build GeoRC, the first benchmark evaluating how well vision-language models (VLMs) can geolocate images. They found LLMS are almost as good at geolocation as the world\u0027s best players, but they struggle to explain their reasoning.\u003C\/p\u003E","format":"limited_html"}],"field_summary_sentence":[{"value":"Researchers have developed what they believe is the strongest benchmark dataset on measuring the geolocation accuracy of LLMs."}],"uid":"36530","created_gmt":"2026-07-06 17:43:39","changed_gmt":"2026-07-06 17:51:06","author":"Nathan Deen","boilerplate_text":"","field_publication":"","field_article_url":"","location":"Atlanta, GA","dateline":{"date":"2026-07-06T00:00:00-04:00","iso_date":"2026-07-06T00:00:00-04:00","tz":"America\/New_York"},"extras":[],"hg_media":{"680557":{"id":"680557","type":"image","title":"_DSC7175.JPG","body":null,"created":"1783359838","gmt_created":"2026-07-06 17:43:58","changed":"1783359838","gmt_changed":"2026-07-06 17:43:58","alt":"Radu Casapu","file":{"fid":"264834","name":"_DSC7175.JPG","image_path":"\/sites\/default\/files\/2026\/07\/06\/_DSC7175.JPG","image_full_path":"http:\/\/hg.gatech.edu\/\/sites\/default\/files\/2026\/07\/06\/_DSC7175.JPG","mime":"image\/jpeg","size":122620,"path_740":"http:\/\/hg.gatech.edu\/sites\/default\/files\/styles\/740xx_scale\/public\/2026\/07\/06\/_DSC7175.JPG?itok=vNYr-PmK"}}},"media_ids":["680557"],"groups":[{"id":"47223","name":"College of Computing"},{"id":"1188","name":"Research Horizons"},{"id":"50876","name":"School of Interactive Computing"}],"categories":[{"id":"194606","name":"Artificial Intelligence"},{"id":"153","name":"Computer Science\/Information Technology and Security"},{"id":"193158","name":"Student Competition Winners (academic, innovation, and research)"}],"keywords":[{"id":"192863","name":"go-ai"},{"id":"187812","name":"artificial intelligence (AI)"},{"id":"187915","name":"go-researchnews"},{"id":"9153","name":"Research Horizons"},{"id":"193556","name":"large language models"}],"core_research_areas":[{"id":"193655","name":"Artificial Intelligence at Georgia Tech"},{"id":"39501","name":"People and Technology"}],"news_room_topics":[],"event_categories":[],"invited_audience":[],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[],"email":[],"slides":[],"orientation":[],"userdata":""}}}