Public Administration Scholarship and the Epidemic of Academic Fraud and Dishonesty Part 3 – The Subversion of Peer Review

The views expressed are those of the author and do not necessarily reflect the views of ASPA as an organization.

By Erik Devereux
December 18, 2023

This column is the third in a series on the epidemic of academic fraud and dishonesty across the social sciences that encompasses public administration (the first column; the second column). Contributing to this scandal is the subversion of peer review, the cherished mechanism of academic scholarship designed to ensure that published research meets acceptable standards. When peer review fails to perform up to design, then fraud and dishonesty are given free reign.

A few years after I became executive director of the Association for Public Policy Analysis and Management (APPAM), University of Maryland Professor Peter Reuter (then editor of the Journal of Policy Analysis and Management) explained to me that blind peer review of research articles was rapidly becoming impossible to implement. The reason in brief is Google. Anyone asked to review a paper for publication easily can cut and paste the abstract into Google’s search bar and instantly be directed to an online version of the same paper with the names and affiliations of the authors included. Authors post their papers online in part to make sure it is found this way. Whereas the authors might never know who reviewed their papers, the reviewers certainly know who the authors are. Personal politics in the review process subsequently becomes impossible to avoid.

Fields connected to the social sciences such as public administration generally do not have available the ultimate arbiter of validity: replication. In some instances, journals can require authors to publish their quantitative data but that alone is not sufficient for replication if the authors have tinkered with the dataset along the lines of, “p hacking.” Very few reviewers will take the time independently to acquire the “raw” data to try to replicate the findings in an article submitted for publication. In the natural sciences, replication not peer review is everything. There are numerous examples across the natural sciences of published research that survived peer review only to be withdrawn when that same research could not be replicated. Cold fusion or hot superconductors, anyone? Relying entirely on peer review already opened the door wide to fraud and dishonesty well before the internal mechanisms of peer review eroded courtesy of Google.

Anyone in academia reading this knows exactly how the game is played today. Researchers in public administration shop their papers around through conferences, meetings and email, all for the purpose of massaging the peer review system in their favor. The APPAM Annual Fall Research Conference that I managed from 1999 to 2009 was a prime example of this shopping exercise. Highly specialized journals keep getting launched every year that have very small circles of academics reviewing each other’s papers. There are many perks to being in such circles. Fueling this is the escalated pressures of academic employment in which “publish or perish” has made the second part of that a constant, ominous looming promise for many.

There is a way to fix peer review but before discussing that I want to emphasize this does not eliminate the underlying causes of academic fraud and dishonesty. Doing that will require major structural changes in higher education to be discussed later in this series.

A consensus is emerging in many social science fields such as psychology that many seminal findings simply cannot be replicated. Out of that has emerged a way to change how journals peer review articles. Instead of sending the finished research out for review, the journal uses a two-step procedure. In the first step, the authors submit the paper up to the point that it discusses the research question(s) and hypotheses, the data to be used, and the specific methodology to be applied to the data. That content is sent for peer review. If accepted, the journal publishes that content and guarantees later publication of the research findings regardless of what those are.

Note how this revision to traditional peer review substantially eliminates the temptations to commit research fraud because university researchers no longer need be concerned that null findings will be rejected for publication. Null findings could continue to lack gravitas in academic hallways but at least they would be published and add to the case for tenure and promotion. This also would overcome the longstanding concern that published research censors a huge amount of vital information by suppressing null findings.

Note that the onus would be heightened on the reviewers to dig into the proposed data and methods rather than gloss over any concerns that might be assuaged by the reported results. In fact, a journal could require that the data be submitted at stage one and that the researchers are held to not changing the data or the methods without going back through another round of peer review.

If you want to hear more about this idea, I strongly recommend this YouTube video.

No one reform is going to cure the epidemic of academic fraud and dishonesty afflicting public administration and other social science fields right now. But the proposed change to how peer review works is a partial vaccine. As we have learned recently, a partial vaccine is better than no vaccine.

---

Author: Erik Devereux is a consultant to nonprofits and higher education and is adjunct faculty in the Security Studies Program at Georgetown University. He has a B.S. from the Massachusetts Institute of Technology (Political Science, 1985) and a Ph.D. from the University of Texas at Austin (Government, 1993). He is the author of Methods of Policy Analysis: Creating, Deploying, and Assessing Theories of Change (available for free here). Email: [email protected]. More content is available here.

(No Ratings Yet)
Loading...