{"655011":{"#nid":"655011","#data":{"type":"event","title":"PhD Defense by Patsorn Sangkloy","body":[{"value":"\u003Cp\u003E\u003Cstrong\u003ETitle: \u003C\/strong\u003EControllable Content Based Image Retrieval and Synthesis\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cstrong\u003EPatsorn Sangkloy\u003C\/strong\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003EPh.D. Student in Computer Science\u003C\/p\u003E\r\n\r\n\u003Cp\u003ESchool of Interactive Computing\u003C\/p\u003E\r\n\r\n\u003Cp\u003EGeorgia Institute of Technology\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cstrong\u003EDate\u003C\/strong\u003E: February 4, 2022\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cstrong\u003ETime\u003C\/strong\u003E: 10:00 AM to 12:00 PM (EST)\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cstrong\u003ELocation \u003C\/strong\u003E(remote via Bluejeans): \u003Ca href=\u0022https:\/\/bluejeans.com\/508230685\u0022\u003Ehttps:\/\/bluejeans.com\/508230685\u003C\/a\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cstrong\u003ECommittee\u003C\/strong\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003EDr. James Hays (Advisor) - School of Interactive Computing, Georgia Institute of Technology\u003C\/p\u003E\r\n\r\n\u003Cp\u003EDr. Devi Parikh - School of Interactive Computing, Georgia Institute of Technology\u003C\/p\u003E\r\n\r\n\u003Cp\u003EDr. Diyi Yang - School of Interactive Computing, Georgia Institute of Technology\u003C\/p\u003E\r\n\r\n\u003Cp\u003EDr. Mark Riedl - School of Interactive Computing, Georgia Institute of Technology\u003C\/p\u003E\r\n\r\n\u003Cp\u003EDr. Subhransu Maji -\u0026nbsp; College of Information and Computer Sciences, University of Massachusetts, Amherst\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cstrong\u003EAbstract\u003C\/strong\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003EA commonly used means to retrieve desired images is by using a text query.\u0026nbsp; This natural form of querying comes with at least two drawbacks. Firstly, the retrieval system may be language specific, limiting its use to only users speaking supported languages. Secondly, there are certain types of target images that would require a lengthy text query to guarantee a successful retrieval. A representative example is an image containing multiple objects at precise locations. The latter drawback is the primary problem we address in this thesis.\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003EIn this thesis, we investigate the use of hand-drawn sketches as a form of query to fetch desired images. Two related but subtly different tasks are studied:\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E1. Content Based Image Retrieval (where target images are retrieved from a database),\u003C\/p\u003E\r\n\r\n\u003Cp\u003E2. Content Based Image Synthesis (where target images are generated).\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003EWe consider two modes of querying:\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E1. Visual content (where a query can be expressed as a simple line drawing sketch, an image patch, or a color scribble),\u003C\/p\u003E\r\n\r\n\u003Cp\u003E2. Language content (where a query can be expressed as a textual description of desired target images).\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003EFor sketch-based image retrieval, we propose a cross-domain network that embeds a user query (sketch) and a target image into a shared feature space, facilitating ready similarity scoring. We collected Sketchy Database; a large-scale dataset of matching sketch and image pairs that can be used as training data.\u0026nbsp; The dataset has been made publicly available and has become one of the few standard benchmarks for sketch-based image retrieval.\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003ETo incorporate both sketch and language content as queries, we propose a late-fusion dual-encoder approach. Our method generalizes CLIP -- a recent successful work on vision and language representation learning -- to sketch-based input. We also collected 5,000 hand-drawn sketches, which can be combined with existing annotated captions in the COCO data to evaluate image retrieval with both text and sketch queries.\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026nbsp;\u003C\/p\u003E\r\n\r\n\u003Cp\u003EFor image synthesis, we present a general framework that allows users to interactively control the generated images based on specification of visual features (e.g., shape, color, texture, sketch). For both retrieval and synthesis tasks, our findings reveal that using a sketch as part of the input makes it easier to succinctly describe desired images.\u003C\/p\u003E\r\n","summary":null,"format":"limited_html"}],"field_subtitle":"","field_summary":"","field_summary_sentence":[{"value":"Controllable Content Based Image Retrieval and Synthesis"}],"uid":"27707","created_gmt":"2022-01-31 21:15:27","changed_gmt":"2022-01-31 21:15:27","author":"Tatianna Richardson","boilerplate_text":"","field_publication":"","field_article_url":"","field_event_time":{"event_time_start":"2022-02-04T10:00:00-05:00","event_time_end":"2022-02-04T12:00:00-05:00","event_time_end_last":"2022-02-04T12:00:00-05:00","gmt_time_start":"2022-02-04 15:00:00","gmt_time_end":"2022-02-04 17:00:00","gmt_time_end_last":"2022-02-04 17:00:00","rrule":null,"timezone":"America\/New_York"},"extras":[],"groups":[{"id":"221981","name":"Graduate Studies"}],"categories":[],"keywords":[{"id":"100811","name":"Phd Defense"}],"core_research_areas":[],"news_room_topics":[],"event_categories":[{"id":"1788","name":"Other\/Miscellaneous"}],"invited_audience":[{"id":"177814","name":"Postdoc"},{"id":"78771","name":"Public"},{"id":"174045","name":"Graduate students"}],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[],"email":[],"slides":[],"orientation":[],"userdata":""}}}