{"462711":{"#nid":"462711","#data":{"type":"news","title":"Assessing the Role of Negative Citations in Science","body":[{"value":"\u003Cp\u003EThe number of times academic articles are cited by subsequent publications is among the time-honored measures used to assess scholarly standing and evaluate academic productivity. But not all of these citations are positive ones, and a paper published this week in the journal \u003Cem\u003EProceedings of the National Academy of Science\u003C\/em\u003Es finds that as many as one in 50 citations in a top immunology journal were critical in nature.\u003C\/p\u003E\u003Cp\u003EThese negative citations may point out limitations, inconsistences or flaws in previous work. The study found that these negative citations were more likely to criticize highly-read papers, and that the criticisms focused on specific sections of the papers. Negative citations tended to originate from scholars who were close to the authors of the original articles in academic discipline and social distance \u2013 but at least 150 miles away geographically.\u003C\/p\u003E\u003Cp\u003EThe research, by authors at the Georgia Institute of Technology, University of Toronto and Massachusetts Institute of Technology, may be the first to systematically quantify and examine these negative citations. The authors hope to expand their study to other disciplines, and believe their work could ultimately lead to a re-examination of how citations are used in academia.\u003C\/p\u003E\u003Cp\u003E\u201cGiven that we rely so heavily on these citation metrics as measures of quality, it\u2019s important to note that the intent of these citations isn\u2019t homogeneous,\u201d said Alexander Oettl, an assistant professor in Georgia Tech\u2019s \u003Ca href=\u0022http:\/\/www.scheller.gatech.edu\/\u0022\u003EScheller College of Business\u003C\/a\u003E and one of the study\u2019s three co-authors. \u201cCriticism is really the cornerstone of the scientific enterprise, yet it\u2019s difficult to get accurate measures of the extent to which criticism of academic papers truly occurs. The negative citations really are different from objective citations.\u201d\u003C\/p\u003E\u003Cp\u003EThe researchers, who also include Nicola Lacetera from the University of Toronto and Christian Catalini from MIT, began with 15,731 full-text articles that had been published in the \u003Cem\u003EJournal of Immunology\u003C\/em\u003E, the top academic journal in that field. From those articles, they extracted 762,355 citations, which referred to 146,891 unique published papers.\u003C\/p\u003E\u003Cp\u003EBeginning with a manual process, they created a set of citations that was used to train a natural language processing program which classified the remainder of the citations as either objective or negative. The process identified 18,304 negative citations, about 2.4 percent of the total citations studied.\u003C\/p\u003E\u003Cp\u003E\u201cThe majority of these negative citations appear to take issue with the \u2018results\u2019 and \u2018discussion\u2019 sections of the papers, so they are really not attempting to overturn theory,\u201d said Oettl. \u201cThey can be attempting to constrain results, note inconsistencies with other research, point out statistical flaws or correct other issues.\u201d\u003C\/p\u003E\u003Cp\u003EIn their study, the authors point out the relative rarity of these negative citations, and speculate that they could either demonstrate a \u201climited, uninfluential role\u201d for criticism in science \u2013 or show a hesitancy to criticize.\u003C\/p\u003E\u003Cp\u003E\u201cMaking these negative citations isn\u2019t without cost,\u201d Oettl noted. \u201cThere could be reputational harm from making these negative citations, and if your criticism turns out to be false, this could heavily impact your reputation within the field. Without this cost, perhaps we might see more overt criticism appearing in the journals.\u201d\u003C\/p\u003E\u003Cp\u003EThe researchers found that while negative citations tended to come from scientists close to the narrow academic topic, the criticism was more likely to involve geographic distance.\u003C\/p\u003E\u003Cp\u003E\u201cWe see that the probability of making a negative citation is much, much lower if you are co-located with the scientist whose work is being critiqued,\u201d Oettl said. \u201cThat potentially speaks to the social component, the social cost of criticism \u2013 you don\u2019t want to criticize someone you may run into on campus. Another possible interpretation is that these issues may be aired face-to-face among scientists who are located near each other geographically.\u201d\u003C\/p\u003E\u003Cp\u003EIn the study, frequently-cited papers were more likely than less-read papers to get negative citations, which stands to reason, Oettl said. \u201cThese are typically more influential papers, so this may have more to do with more people reading them and more of an incentive for scholars to take issue with important papers,\u201d he explained. \u201cFurthermore, less important papers may receive less scrutiny, as pointing out limitations and shortcomings of trivial work will not drastically shift the scientific frontier.\u201d\u003C\/p\u003E\u003Cp\u003EReplication of previous work helps ensure the accuracy of research, and criticism of published studies can lead to correction that makes science more robust. But does the current scientific publishing system encourage enough discussion?\u003C\/p\u003E\u003Cp\u003E\u201cThe pessimistic view is that only one in 50 citations is negative in nature, so possibly there isn\u2019t as much debate as we would want for a healthy discussion,\u201d Oettl said. \u201cBut on the other hand, by the time a major manuscript makes it into print, it has undergone a tremendous amount of criticism. We now have some evidence of the extent to which criticism, in the form of negative citations, occurs within scientific manuscripts.\u201d\u003C\/p\u003E\u003Cp\u003EThe researchers chose immunology because they had access to the papers, which were consistent in format. They found similar numbers of negative citations in a small set of mathematics-related papers, but hope to expand their work to determine if their findings apply to other disciplines. Ultimately, they hope the work could change the way citations are used in science.\u003C\/p\u003E\u003Cp\u003E\u201cWe wanted to put forward a methodology to help future scholars think about how we can possibly expand this classification infrastructure and attract additional metadata to the citations that we make,\u201d Oettl said. \u201cOur paper is very descriptive, but it identifies something that nobody has really demonstrated before: that these negative citations behave very differently from others.\u201d\u003C\/p\u003E\u003Cp\u003EThough the current study focused on a single discipline, immunology, \u201cwe would really like to see this work and methodology expanded to other disciplines,\u201d Oettl said. \u201cUltimately, though, we care about what role these negative citations play in aiding scientific debate and in the advancement of the scientific frontier.\u201d\u003C\/p\u003E\u003Cp\u003EThe paper was published October 26 in the journal\u0027s Early Edition.\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003ECITATION\u003C\/strong\u003E: Christian Catalini, Nicola Lacetera, Alexander Oettl, \u201cBeyond citation counts: Exploring negative citations in science,\u201d (Proceedings of the National Academy of Sciences, 2015). \u003Ca href=\u0022http:\/\/www.dx.doi.org\/10.1073\/pnas.1502280112\u0022\u003Ehttp:\/\/www.dx.doi.org\/10.1073\/pnas.1502280112\u003C\/a\u003E.\u003C\/p\u003E\u003Cp\u003E\u003Cbr \/\u003E\u003Cstrong\u003EResearch News\u003C\/strong\u003E\u003Cbr \/\u003E\u003Cstrong\u003EGeorgia Institute of Technology\u003C\/strong\u003E\u003Cbr \/\u003E\u003Cstrong\u003E177 North Avenue\u003C\/strong\u003E\u003Cbr \/\u003E\u003Cstrong\u003EAtlanta, Georgia 30332-0181 USA\u003C\/strong\u003E\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EMedia Relations Contact\u003C\/strong\u003E: John Toon (\u003Ca href=\u0022mailto:jtoon@gatech.edu\u0022\u003Ejtoon@gatech.edu\u003C\/a\u003E) (404-894-6986)\u003C\/p\u003E\u003Cp\u003E\u003Cstrong\u003EWriter\u003C\/strong\u003E: John Toon\u003C\/p\u003E","summary":null,"format":"limited_html"}],"field_subtitle":"","field_summary":[{"value":"\u003Cp\u003EA new study conducted by researchers from three institutions has examined in the role of negative citations in academic publishing. The researchers found that one in 50 citations from a major academic journal contained criticism of previous work.\u003C\/p\u003E","format":"limited_html"}],"field_summary_sentence":[{"value":"A new study examines the role of negative citations in academic publishing."}],"uid":"27303","created_gmt":"2015-10-26 16:13:13","changed_gmt":"2016-10-08 03:19:51","author":"John Toon","boilerplate_text":"","field_publication":"","field_article_url":"","dateline":{"date":"2015-10-26T00:00:00-04:00","iso_date":"2015-10-26T00:00:00-04:00","tz":"America\/New_York"},"extras":[],"hg_media":{"462801":{"id":"462801","type":"image","title":"Alexander Oettl crop","body":null,"created":"1449256373","gmt_created":"2015-12-04 19:12:53","changed":"1475895171","gmt_changed":"2016-10-08 02:52:51","alt":"Alexander Oettl crop","file":{"fid":"202942","name":"oett_crop.jpg","image_path":"\/sites\/default\/files\/images\/oett_crop_0.jpg","image_full_path":"http:\/\/hg.gatech.edu\/\/sites\/default\/files\/images\/oett_crop_0.jpg","mime":"image\/jpeg","size":196387,"path_740":"http:\/\/hg.gatech.edu\/sites\/default\/files\/styles\/740xx_scale\/public\/images\/oett_crop_0.jpg?itok=55kP4uKK"}},"462671":{"id":"462671","type":"image","title":"Alexander Oettl","body":null,"created":"1449256373","gmt_created":"2015-12-04 19:12:53","changed":"1475895209","gmt_changed":"2016-10-08 02:53:29","alt":"Alexander Oettl","file":{"fid":"203645","name":"oettl_alex_atrium_xlrg.jpg","image_path":"\/sites\/default\/files\/images\/oettl_alex_atrium_xlrg_1.jpg","image_full_path":"http:\/\/hg.gatech.edu\/\/sites\/default\/files\/images\/oettl_alex_atrium_xlrg_1.jpg","mime":"image\/jpeg","size":1563254,"path_740":"http:\/\/hg.gatech.edu\/sites\/default\/files\/styles\/740xx_scale\/public\/images\/oettl_alex_atrium_xlrg_1.jpg?itok=Bvhlej2n"}},"462691":{"id":"462691","type":"image","title":"Scheller College - Tech Square","body":null,"created":"1449256373","gmt_created":"2015-12-04 19:12:53","changed":"1475895209","gmt_changed":"2016-10-08 02:53:29","alt":"Scheller College - Tech Square","file":{"fid":"203647","name":"technology-square.jpg","image_path":"\/sites\/default\/files\/images\/technology-square_0.jpg","image_full_path":"http:\/\/hg.gatech.edu\/\/sites\/default\/files\/images\/technology-square_0.jpg","mime":"image\/jpeg","size":1995590,"path_740":"http:\/\/hg.gatech.edu\/sites\/default\/files\/styles\/740xx_scale\/public\/images\/technology-square_0.jpg?itok=UVygy2fs"}}},"media_ids":["462801","462671","462691"],"groups":[{"id":"1188","name":"Research Horizons"}],"categories":[{"id":"139","name":"Business"},{"id":"135","name":"Research"}],"keywords":[{"id":"145771","name":"academic publishing"},{"id":"52911","name":"Alexander Oettl"},{"id":"7581","name":"citation"},{"id":"97961","name":"criticism"},{"id":"3230","name":"journal"},{"id":"145761","name":"negative citation"},{"id":"167089","name":"Scheller College of Business"}],"core_research_areas":[],"news_room_topics":[{"id":"71901","name":"Society and Culture"}],"event_categories":[],"invited_audience":[],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[{"value":"\u003Cp\u003EJohn Toon\u003C\/p\u003E\u003Cp\u003EResearch News\u003C\/p\u003E\u003Cp\u003E\u003Ca href=\u0022mailto:jtoon@gatech.edu\u0022\u003Ejtoon@gatech.edu\u003C\/a\u003E\u003C\/p\u003E\u003Cp\u003E(404) 894-6986\u003C\/p\u003E","format":"limited_html"}],"email":["jtoon@gatech.edu"],"slides":[],"orientation":[],"userdata":""}}}