Dismissive Reviews, Citation Cartels, and the Replication Crisis.pptx

This interdisciplinary theme of the Conference addresses two of the very serious and controversial challenges of modern-day research, namely dismissive reviews (unsupported declarations of scholars declaring no previous research exists on certain topics, despite evidence of the contrary) and „citation cartels” (groups of scholars who agree to cite each others’ work and none of the other available literature). These practices have a considerable impact in the quality and ethical
aspects of research, and are also reflected in the replicability crisis.

1. Dismissive Reviews, Citation Cartels, and the Replication Crisis Richard P. Phelps

2. The Replication Crisis

3. … smaller the sample size. … smaller the effect. … greater the number of tested relationships. … greater the flexibility in design, definitions, outcomes, and analytical modes. … greater the financial and other interests and prejudices. … hotter the scientific field. … greater the publicity. According to John Ioannidis … … research findings are less likely to be true, the …

4. Other researchers found “gaming the system” to be widespread “P-hacking” Only requiring P < .05 Non-replicable findings Many scholars refusing to share data sets I add 2 more: Dismissive Reviews, Citation Cartels Ioannidis exaggerated, but much of what he wrote was true

5. Proportion of studies replicated at p < .05, in three fields of study, 2015 to 2018

6. Number of citations to replicated and not replicated studies, as of 2021

7. The growth of “Metascience” as a field of study Research Policy (journal), 1971 Scientometrics (journal), 1978 Cochrane Library, 1993 Metascience (journal), 1996 Campbell Collaboration, 2000 Research Ethics (journal), 2005 Centre for Journalology, 2005 Journal of Informetrics , 2007 Retraction Watch, 2010 PubPeer, 2012 Science Advances (journal), 2015 Research Waste/EQUATOR, 2015 Research Integrity and Peer Review (journal), 2016

8. Information proliferation Pro-Quest UMI dissertation publishing: 3 + 5 million dissertations and theses 250,000 new works each year STM reports : 34,000 journals +4%/year 10,000 publishers

9. Still more proliferation As of 2021… …3 million articles are published in peer-reviewed journals every year. More than 5 new articles per minute.

10. The proliferation of researchers Residing in the United States alone (2008): 2.5 million with doctoral degrees 5.5 million with professional degrees 14.9 million with masters’ degrees

11. Knowing ALL the research literature on a topic There is so much, is anyone qualified to speak for all of it? It is genuinely difficult to do something new and unique

12. Knowledge is Unlimited? It may be, but there are limits to the amount that we can use. So, we filter it. Two ways to filter: Summarize all of it Accept only a certain amount, a certain type, …or only from certain people

13. The World Wide Web is an information filter 20 th Century: Reference librarians provides wide range of information 21 st Century: Search Engine Optimization (SEO) narrows range

14. Definitions: Firstness Claims & Dismissive Reviews With a firstness claim, a researcher insists that s/he is the first to study a topic. With a dismissive literature review, a researcher assures the reader that no one else has conducted a study on a topic.

15. The Effect of Firstness Claims and Dismissive Reviews The public is told that no other research exists on a topic, ergo, there is no reason to look for it.

16. How difficult is a literature review? Not analytically difficult But a genuinely thorough review requires a substantial amount of time, and some money Generally, neither search nor ethics are part of US professors’ training.

17. Professional incentives to do a thorough literature review THERE ARE NONE? Scholars get little credit for a thorough literature review, much more for “ original work ” In “ publish or perish ” environments, lit reviews are impediments to progress

18. Why do a thorough lit review? huge burden in time and distraction little to no benefit professionally no punishment for not doing it

19. Dismissive Reviews and Citation Cartels in Education Policy Research – A List https:// nonpartisaneducation.org /Review/Resources/ DismissiveList.htm A database of > 1,000 dismissive reviews from top US scholars in education policy 3 types: Firstness Claims Dismissals Denigrations

20. Examples of Firstness Claims “This report is unique.” “This is the only study to …” “We provide the first evidence …” “We construct the first nationwide measures of …” “Our study is the first to precisely estimate …” “We are the first to isolate the impact of …” “Ours is the first comprehensive analysis of …” “To our knowledge, this has not been studied before …”

21. Examples of Dismissals “There is very little research …” “Surprisingly little is known about …” “The debate is mostly supported by anecdotes.” “Scant empirical evidence has been provided.” “Studies of … are only relatively recent.” “Knowledge of … has remained nonexistent.” “There has been little systematic effort to study …”

22. Examples of Denigrations “Previous work is very limited …” “To provide more rigorous evidence, we …” “Existing studies focus on a very small number of ….” “[we] found the bulk of studies to be flawed …” “… significant measurement problems in previous studies.” “[earlier] studies were unable to control for ....” “[earlier] studies had major deficiencies …” “There are statistical shortcomings in … the studies.”

23. “absence of evidence is not evidence of absence” Eyewitness fallacy – what you see represents all Composition fallacy – the whole must have a property because its parts have the property. Amazing Familiarity – the speaker seems to have information that there is no possible way for him to get Argumentum ad ignorantiam – a proposition is true if it has not been proven false, or false if not proven true. Wishful Thinking – things are (or will be) some way because that is how we wish them to be.

24. Raw Declaration No evidence of a literature search provided Dismissed Work Not Identified Providing no help in finding it Discouraging search for it Avoiding debate Key Characteristics of Dismissive Reviews

25.   “absence of research” ~33,000 “absence of studies” ~26,000 “this is the first study” ~1,610,000 “little research” ~931,000 “paucity of research” ~121,000 S earch engine counts (Google Scholar), 9 .5.2023

26. Dismissive reviews worse than plagiarism Misrepresent the work of one person (by plagiarizing) reward is small (saves some work & time) risk is large (could ruin one ’ s reputation and career) Misrepresent the work of hundreds (in dismissive reviews) reward is large (for being first & unopposed) risk is nil

27. Ethics of dismissive reviews “ Whatever you allow, you encourage. ” Michael Josephson

28. Definition: Citation Cartels In a Citation Cartel, scholars… … cite and reference each other and … ignore, dismiss, or denigrate other research.

29. Definition: “Sincere” Scholarship Sincere scholars behave ethically : C onduct forthright and thorough literature reviews, C ite all other, relevant research regardless of: their own personal opinions of the individual scholars their own personal preferences for their research results. They provide full and accurate citations of all that previous work, so that readers will have no trouble finding it. Focus is on accuracy and adding knowledge 

30. Definition: “Strategic” Scholarship Strategic scholars behave strategically : Avoid literature searches and reviews Either: Declare ”firstness” – no previous research Reference only work from within their cartel Do not pass up opportunities to promote their work… …even if they lack expertise on a topic, they will say something Focus is on advancing their careers

31. Comparing Strategic and Sincere Scholarship: Citation accumulation over thirty years

32. How can they do this? No legal restrictions on education research quality – no Hippocratic Oath, no license or certification Rewards still lie within one’s primary field – economics for economists, psychology for psychologists, etc. Scholars are rewarded for what they add; but they are not punished for what they subtract Many who could “blow the whistle” are afraid, or they might dislike the research literature and like dismissing it

33. Journalists help to suppress information When they print one researcher ’ s firstness claim or dismissive review, they help to suppress others ’ work and competing evidence

34. * saves time, avoids tedium of reading the research literature * adds to own citation totals, or those of one’s citation cartel, … while not adding to rivals’ citation totals * gives readers no help in finding rival evidence (by not even citing it) * establishes (false) bona fides as an “expert” on the topic * attention by allegedly being "first," "original," "a pioneer." * increases the likelihood of press coverage for the same reason. * increases prospects for grant funding to "fill knowledge gaps." Benefits accrue to… …individual scholars and small groups:

35. Costs accrue to… society Society loses information; remaining information is skewed in favor of the powerful Policy decisions are based on information that is limited and skewed Government and foundations may pay again for research that has already been done

36. Research most vulnerable to dismissal That done by those below the “ celebrity threshold ” * Studies by civil servants (government agencies do not promote or defend their work) That done by the deceased All become: “ Zombie Researchers ” * Researchers below the celebrity threshold lack the resources and media access to successfully counter dismissals of their work – they can easily be ignored.

37. Paradox of research proliferation As the amount of research grows… …so does the amount declared nonexistent …so does the incentive to dismiss it …so does the opportunity to dismiss it

38. What Can be Done?

39. Transparency Too easy to publish dismissive reviews: Find 1 editor and 1-2 reviewers among the hundreds of thousands of journals available If first journal one tries objects, there are many others. Just submit same paperwork to another journal. A genuine lit review takes weeks or months. Preprints and PubPeer are the only current, effective deterrents

40. Ban firstness claims and dismissive reviews Add ban to the ethics codes of… …journalists …foundation research funders …government research funders In most cases, editors, reviewers, & journalists have neither the time nor the resources to verify

41. Real punishment for false firstness claims and dismissive reviews Make literature reviews optional for getting funding, but… …make their accuracy mandatory, …suspend violators from any further funding

42. Remove any literature review obligation from research articles Removes temptation Most do more harm than good anyway because they are partial and selective

43. Is meta-analysis the solution? Let meta-analysts do all literature reviews Meta-analysis review model: Identify where you have looked before making summary claims

44. Merci !