I confess, I wrote the Arsenic DNA paper to expose flaws in peer-review at subscription based journals

In 2011, after having read several really bad papers in the journal Science, I decided to explore just how slipshod their peer-review process is. I knew that their business depends on publishing “sexy” papers. So I created a manuscript that claimed something extraordinary – that I’d discovered a species of bacteria that uses arsenic in its DNA instead of phosphorus. But I made the science so egregiously bad that no competent peer reviewer would accept it. The approach was deeply flawed – there were poor or absent controls in every figure. I used ludicrously elaborate experiments where simple ones would have done. And I failed to include a simple, obvious experiment that would have definitively shown that arsenic was really in the bacteria’s DNA. I then submitted the paper to Science, punching up the impact the work would have on our understanding of extraterrestrials and the origins of life on Earth in the cover letter. And what do you know? They accepted it!

My sting exposed the seedy underside of “subscription-based” scholarly publishing, where some journals routinely lower their standards – in this case by sending the paper to reviewers they knew would be sympathetic – in order to pump up their impact factor and increase subscription revenue. Maybe there are journals out there who do subscription-based publishing right – but my experience should serve as a warning to people thinking about submitting their work to Science and other journals like it. 

OK – this isn’t exactly what happened. I didn’t actually write the paper. Far more frighteningly, it was a real paper that contained all of the flaws described above that was actually accepted, and ultimately published, by Science.

I am dredging the arsenic DNA story up again, because today’s Science contains a story by reporter John Bohannon describing a “sting” he conducted into the peer review practices of open access journals. He created a deeply flawed paper about molecules from lichens that inhibit the growth of cancer cells, submitted it to 304 open access journals under assumed names, and recorded what happened. Of the 255 journals that rendered decisions, 157 accepted the paper, most with no discernible sign of having actually carried out peer review. (PLOS ONE, rejected the paper, and was one of the few to flag its ethical flaws).

The story is an interesting exploration of the ways peer review is, and isn’t, implemented in today’s biomedical publishing industry. Sadly, but predictably, Science spins this as a problem with open access. Here is their press release:

Spoof Paper Reveals the “Wild West” of Open-Access Publishing

A package of news stories related to this special issue of Science includes a detailed description of a sting operation — orchestrated by contributing news correspondent John Bohannon — that exposes the dark side of open-access publishing. Bohannon explains how he created a spoof scientific report, authored by made-up researchers from institutions that don’t actually exist, and submitted it to 304 peer-reviewed, open-access journals around the world. His hoax paper claimed that a particular molecule slowed the growth of cancer cells, and it was riddled with obvious errors and contradictions. Unfortunately, despite the paper’s flaws, more open-access journals accepted it for publication (157) than rejected it (98). In fact, only 36 of the journals solicited responded with substantive comments that recognized the report’s scientificproblems. (And, according to Bohannon, 16 of those journals eventually accepted the spoof paper despite their negative reviews.) The article reveals a “Wild West” landscape that’s emerging in academic publishing, where journals and their editorial staffs aren’t necessarily who or what they claim to be. With his sting operation, Bohannon exposes some of the unscrupulous journals that are clearly not based in the countries they claim, though he also identifies some journals that seem to be doing open-access right.

Although it comes as no surprise to anyone who is bombarded every day by solicitations from new “American” journals of such-and-such seeking papers and offering editorial positions to anyone with an email account, the formal exposure of hucksters out there looking to make a quick buck off of scientists’ desires to get their work published is valuable. It is unacceptable that there are publishers – several owned by big players in the subscription publishing world – who claim that they are carrying out peer review, and charging for it, but no doing it.

But it’s nuts to construe this as a problem unique to open access publishing, if for no other reason than the study, didn’t do the control of submitting the same paper to subscription-based publishers (UPDATE: The author, Bohannon emailed to say that, while his original intention was to look at all journals, practical constraints limited him to OA journals, and that Science played no role in this decision). We obviously don’t know what subscription journals would have done with this paper, but there is every reason to believe that a large number of them would also have accepted the paper (it has many features in common with the arsenic DNA paper afterall). Like OA journals, a lot of subscription-based journals have businesses based on accepting lots of papers with little regard to their importance or even validity. When Elsevier and other big commercial publishers pitch their “big deal”, the main thing they push is the number of papers they have in their collection. And one look at many of their journals shows that they also will accept almost anything.

None of this will stop anti-open access campaigners  (hello Scholarly Kitchen) from spinning this as a repudiation for enabling fraud. But the real story is that a fair number of journals who actually carried out peer review still accepted the paper, and the lesson people should take home from this story not that open access is bad, but that peer review is a joke. If a nakedly bogus paper is able to get through journals that actually peer reviewed it, think about how many legitimate, but deeply flawed, papers must also get through. Any scientist can quickly point to dozens of papers – including, and perhaps especially, in high impact journals – that are deeply, deeply flawed – the arsenic DNA story is one of many recent examples. As you probably know there has been a lot of smoke lately about the “reproducibility” problem in biomedical science, in which people have found that a majority of published papers report facts that turn out not to be true. This all adds up to showing that peer review simply doesn’t work.

And the real problem isn’t that some fly-by-night publishers hoping to make a quick buck aren’t even doing peer review (although that is a problem). While some fringe OA publishers are playing a short con, subscription publishers are seasoned grifters playing a long con. They fleece the research community of billions of dollars every year by convincing them of something manifestly false – that their journals and their “peer review” process are an essential part of science, and that we need them to filter out the good science – and the good scientists – from the bad. Like all good grifters playing the long con, they get us to believe they are doing something good for us – something we need. While they pocket our billions, with elegant sleight of hand, then get us to ignore the fact that crappy papers routinely get into high-profile journals simply because they deal with sexy topics.

But unlike the fly by night OA publishers who steal a little bit of money, the subscription publishers’ long con has far more serious consequences. Not only do they traffic in billions rather than thousands of dollars and denying the vast majority of people on Earth access to the findings of publicly funded research, the impact and glamour they sell us to make us willing participants in their grift has serious consequences. Every time they publish because it is sexy, and not because it is right, science is distorted. It distorts research. It distorts funding. And it often distorts public policy.

To suggest – as Science (though not Bohannon) are trying to do – that the problem with scientific publishing is that open access enables internet scamming is like saying that the problem with the international finance system is that it enables Nigerian wire transfer scams.

There are deep problems with science publishing. But the way to fix this is not to curtain open access publishing. It is to fix peer review.

First, and foremost, we need to get past the antiquated idea that the singular act of publication – or publication in a particular journal – should signal for all eternity that a paper is valid, let alone important. Even when people take peer review seriously, it is still just represents the views of 2 or 3 people at a fixed point in time. To invest the judgment of these people with so much meaning is nuts. And its far worse when the process is distorted – as it so often is – by the desire to publish sexy papers, or to publish more papers, or because the wrong reviewers were selected, or because they were just too busy to do a good job. If we had, instead, a system where the review process was transparent and persisted for the useful life of a work (as I’ve written about previously), none of the flaws exposed in Bohannon’s piece would matter.

This entry was posted in open access, science. Bookmark the permalink. Both comments and trackbacks are currently closed.


  1. Posted October 3, 2013 at 11:52 am | Permalink

    Re Science’s Fake Paper sting: for me, one aspect of this deplorable business stood out in particular – the question: Are all Open Access Journals created equal? The answer to that would seem to be an obvious ‘No’, especially given the outcome of this sting. But then it would beg the follow-up question, if this had indeed been a serious and genuine paper, would the author (say, Bohannon) seek out obscure OA journals for publishing it? If yes, to what end – simply padding one’s resumé by increasing the publication record? Are people who see and evaluate resumés for professional purposes really fooled by that?

    Rather than criticizing the Open Access publishing process, the most obvious solution – again, to me – seems to be to institute a measure of quality assessment for Open Access journals. I am not an expert in the publishing business, but surely some kind of reasonable and workable metric can be worked out in the same way Thomson Reuters did all those years ago for Pay-for-Play journals?

    At the same time, one must acknowledge that the Peer Review system is not a magic wand that can separate the grain from the chaff by a simple touch. I mean, look at the thriving Elsevier Journal Homeopathy, allegedly peer reviewed… Has that ever stemmed the bilge it churns out regularly?

    But the other question that really, really bothers me is more fundamental: how and when did India become this haven for dubious, low quality Open-Access publishing? (For the context, see this interactive map of the sting.)

    • Posted October 3, 2013 at 1:01 pm | Permalink

      There is a simple solution to quality control for peer review in open access journals: open peer review. This same issue of Science features an interview with Vitek Tracz, about F1000Research’s open peer review system. We include all peer reviewer names and their comments with all papers, so you can see exactly who looked at a paper and what they said.

      • Jade17
        Posted October 6, 2013 at 12:56 am | Permalink

        I disagree. What good would having the names of the reviewers do? It would give the authors a chance to chew out people with poor reviews of their paper, or ask for clarification on comments…but is that going to improve the peer review process? No. To improve the process, I think all submissions should be completely anonymous. Emails can be encrypted (think Craigslist) through the submission system online, and names eliminated from the paper. This way well-known authors receive no preferential treatment, and the focus is on the quality of the work rather than the name or institution associated with it.

        • Derek B
          Posted October 6, 2013 at 12:47 pm | Permalink

          Open peer review is what we want, where people can’t half-ass and hide behind anonymity. Science is about reputation within the community, and someone using bad methodology or allowing their metaphysical assumptions to skew the way they publish results or conduct experiments will get caught and black-listed quickly, whereas it might take years of complaints to publishers to weed them out on the back end.

          Open peer review will only work if ALL data are published, not just the summary or what the researchers felt was “important”. This is one of the reasons that we see so many experiments that are non-replicable.

          Science is about collaboration. Collaboration is only possible if we can get a big picture and everyone is on equal footing, including that they think twice about jumping on something they don’t like, because their name and reputation are attached to their critiques. Other systems have used this type of approach to quickly weed out bad players and encourage good ones (hell, reviewing peers could even capitalize on their skill and expertise by building a reputation and open peer review mags could hire them to conduct review, but only after the community and that individual have demonstrated their prowess and integrity). I want to see the ebay and amazon of peer review where 99.9% of all “transactions” are beneficial and produce good results. It’s coming, it’s now a matter of when, and not if.

        • George
          Posted October 7, 2013 at 12:04 pm | Permalink

          I agree Jade17. I’ve seen high ranking authors getting their way easily through the “review” process. I agree that making the whole process – including journal, authors, and reviewers – secret, it would make the reviewers more objective.
          On the other hand, making the reviewers visible will probably lead the big ones patting each other and challening the younger reviewers off-stage…

        • Andrea Campisano
          Posted October 10, 2013 at 12:49 pm | Permalink

          I agree with this. It’s not so much anonymous reviewers, as it it a matter of anonymous authors…

          Although a smart person in the field will know who is likely to be the author, after reading, so back to square 1.

        • Ray Vazquez
          Posted October 13, 2013 at 12:34 pm | Permalink

          F1000 is a joke too, the are completely biased by high impact publications and there is no sign that they actually read any paper, they actually sense the media and then simply agree with the buzz, nonsensical.

          • hypatia
            Posted May 2, 2014 at 8:08 am | Permalink

            I agree – F1000 is a joke. Just trying to capture as much free content as possible (with posters, slides, etc.) without any effort at quality control. And they are not very honest about their submission process – try requesting them to remove a submission and see what happens if anything. A business built on public funding and free labor – how much more exploitative can one get? The whole OA thing has become a business more than a real solution facing academic publishing, which can only be tackled effectively by the reward system and academics themselves. This is the tragic state of science and the corruption of scientists today.

        • Chabaka
          Posted January 13, 2014 at 9:31 pm | Permalink

          I fully agree with Jade17. I saw a paper (ab initio computation in chemistry) submitted with the name of a Nobel prize winner (experimental chemist, who worked in the same field) attached to it, which contained lots of typos, mistakes, wrong figuresd and what not, and yet the paper getting accepted and published, even after one referee rejected it twice.

    • John
      Posted October 3, 2013 at 4:43 pm | Permalink

      Congratulations Kausik, you just misused the phrase “begs the question”. Check yourself before you wreck yourself.

      • cube
        Posted October 4, 2013 at 5:35 am | Permalink

        ‘cuz misusin’ a phrase is bad for ya health.

        (sorry, couldn’t resist)

      • Jose Jiminez
        Posted October 4, 2013 at 8:08 am | Permalink

        Yeah, a common misuse of a common phrase pretty much nullifies absolutely everything else.

    • GhoshBabu
      Posted October 5, 2013 at 11:10 am | Permalink

      It all depends on the nature of publisher. Bad guys sent their papers to dubious publishers whereas I always prefer Nature, Science or PLoS.

  2. DrugMonkey
    Posted October 3, 2013 at 12:06 pm | Permalink

    The fact that peer review is not as capable as some would like to assume or as some actively claim does not make it a “joke”.

    • Derek B
      Posted October 6, 2013 at 12:50 pm | Permalink

      The scale of its documented failures over the past 10 years (the sheer number of non-replicable tests and experiments within the literature, of late, is staggering, especially considering that honesty is one of the foundational ethical assumptions of the true scientific method) is no joking matter.

      get rid of anonymous peer review and you get rid of a lot of this problem, because every reviewer’s reputation is on the line, and reputations acts as a very strong, yet passive, incentive for people to not impose metaphysical presuppositions or ethical preferences into data. We are seeing way too much of that kind of junk these days.

      • George Sangster
        Posted October 15, 2013 at 7:58 pm | Permalink

        Getting rid of anonymous peer review is also important because it forces referees to be more constructive (it will expose unnecessarily negative reviews by ‘pitbull referees’).

        On the other hand, without anonymity, potential reviewers might either soften their criticism to avoid backlash from the authors, or simply decline to review. Many ‘regular’ journals already struggle with finding expert referees within a reasonable time, and this will get worse if referees feel no longer protected by anonymity.

      • Curtis Bennett
        Posted March 26, 2014 at 9:42 am | Permalink

        How many junior faculty would willingly do a negative peer-review of a well-known authority in their field that might be sitting on grant committees if the name of the reviewer was attached?

        • CRF
          Posted May 22, 2014 at 12:57 pm | Permalink

          Well obviously that is another serious problem in the world of science – but it’s also nothing new.

  3. Posted October 3, 2013 at 12:36 pm | Permalink

    Thank you for putting Science‘s pathetic hatchet-job into perspective. Whatever respect I might once have had for their racket is rapidly dwindling to nothing. Contemptible.

  4. Posted October 3, 2013 at 12:53 pm | Permalink

    Anonymous peer review by 2-3 reviewers whose opinion may or may not be respected by the journal editor, is the perfect system to sustain a journal monopoly over scientific evaluation and thus secure revenue for commercial publishers. It is however the worst system for everyone else: authors, reviewers, science and society.

    The solution is simple:
    1. Publish your article immediately in institutional repositories or preprint servers
    2. Engage an open peer review process, independent of journals and guided by authors themselves
    3. Improve your article based on formal reviews by peers and commentary by the wider community
    4. Whenever you feel your article has been evaluated by a sufficient number of experts and all critics have been adequately addressed submit your work to a journal of your choice

    As Michael intelligently notes, the views of 2-3 anonymous people at a fixed point in time —and who receive no credit for their work— cannot be the cornerstone of scientific evaluation. By definition science is dynamic and advances faster and more efficiently when scientists openly collaborate instead of competing for a limited number of publication slots in “prestigious” academic journals.

    • Posted October 3, 2013 at 1:41 pm | Permalink

      Or maybe just skip step 4. Of course, that’s problematic if you’re an academic whose livelihood depends on racking up academic brownie points. But if, like me, you’re free from all that (I left academia and make my money from software), I see little reason to bother with journals anymore. No offense, Mike. PLoS is a big improvement on rackets like Science. But I doubt many scientists, or at least many good ones, read things on the basis of where they’re published these days. And to the extent people follow step 1, work is old news by the time it appears in a journal anyhow. That’s why at this point, I don’t expect to publish in a journal ever again (except when I’m a coauthor and another author needs the brownie points).

      • Will
        Posted October 5, 2013 at 6:02 am | Permalink

        “Of course, that’s problematic if you’re an academic whose livelihood depends on racking up academic brownie points.”

        I think that’s a fundamental problem that needs to be tackled before the issue of crummy peer review and sexy publishing for high impact can be dealt with. The fact that one’s career may very well depend on publishing something — anything — in what is deemed to be a “sexy” journal creates a push that prevents meaningful reform. It would be great to be able to skip step 4 but how many people feel like their hands are tied in this process?

        • librarian
          Posted October 15, 2013 at 11:18 am | Permalink

          Agreed. If we follow that model and go through step 4, then why are we saddling our libraries with million dollar (or more) annual subscription fees for work that has already been openly vetted by the community and is freely available for others to build upon? it seems like it is very much time to start asking where our resources in academe are going and if supporting this kind of publishing market is really benefiting us or just serving to hold up an antiquated tenure system that needs to take a hard look at it’s evaluation system.

    • Justin Ashworth
      Posted October 4, 2013 at 7:51 am | Permalink

      [Re: open self-publishing (Perakakis)]
      What are some good successful examples of things like this having been done? Clearly the technology exists to submit the entirety one’s work to open scrutiny in very accessible and free formats regardless of journal policies (that technology being: the webserver.) Who’s going to actually do it prior to journal submission? Is someone already doing it? Or who will be the first to exemplify this (successsfully?) Most of us doing integrative or data-heavy research are already self-publishing large amounts of data, analysis, results and tools online that far exceed the scope and formats of journal articles (often with sincere hopes that a couple of too busy reviewers will use them to judge a submitted paper.) Some of the reasons that we don’t make these resources immediately public at inception (pre-publication) are a) secrecy, b) competition, c) time advantage, d) agreements with others, e) lack of mature web implementation or f) we simply don’t have to. None of these reasons seem very good though, unless of course you get scooped on a [Science] Paper by tipping off or clueing in your competition, or someone criticizes the work so badly and publically (perhaps even unfairly) that it spoils the chances of getting credit (+jobs and funding) for your work. How do you change the game so that the incentives to self-publish (and solicit public review) outweigh the risks? Perhaps it makes sense for established elites (or large consortia) who are relatively insulated from the risks to set a good example.

      Also on the main subject–funny that one need look no further than Bohannon’s piece itself for an example of misconmmunicated and poorly reviewed research being published in an elite journal. 🙂

      • Posted October 4, 2013 at 10:51 am | Permalink

        Re: Justin

        Actually, aren’t there more risks involved in submitting your manuscript to a journal editor who will send it to anonymous experts (should I say competitors) in your field who are free to “bury” your work without any accountability? Self-publishing lets you license and register your work immediately, while you can continue improving the manuscript and upload new versions. Inviting peers to openly assess your work is a further step that can help improve a paper by uncovering weak points and limitations, and addressing critiques by expert colleagues. This actually should give you a better chance of publishing your manuscript to a better journal. But, to be honest, what we aspire is to create an alternative open and transparent system of scientific evaluation immune to the existing conflicts of interests between authors, reviewers, journal editors and publishers.
        For more feel free to have a look at our organization’s website.

      • librarian
        Posted October 15, 2013 at 11:23 am | Permalink

        this is over a year old, but it’s one of the best descriptions of an OA model out there. it is not prohibitively expensive or overly burdensome to take back publishing in our institutions. I encourage everyone here to take this issue up with your library – they may already have a product in place that can help manage publishing journals in your field.

  5. Posted October 3, 2013 at 1:03 pm | Permalink

    @ Drug Monkey – that fact alone is not sufficient, you’re right. But it does suggest that peer review is not the pinnacle of virtue that journals claim it is, and use as justification for their continued extortion for subscription fees.

  6. Mitch Skinner
    Posted October 3, 2013 at 1:04 pm | Permalink

    The author, Bohannon emailed to say that, while his original intention was to look at all journals, practical constraints limited him to OA journals, and that Science played no role in this decision

    Really? Science may not have explicitly asked him to restrict his attention to OA journals, but he is undoubtedly fully aware of how his bread is buttered. Asking us to believe that the market for his story played no role in is reporting decisions is a bit much. As with any extraordinary claim, he needs to back that up with more than just a bare assertion if he expects anyone to actually buy it.

    • David Roos
      Posted October 6, 2013 at 12:42 am | Permalink

      As noted in his article, Bohannon’s inquiry was initiated in response to an email from me about a Nigerian student who fell prey to a predatory publisher. I was not aware of (and played no part in) the investigation, but find it plausible that this sting operation was developed to focus on the prevalence and quality of peer review in predatory journals, and then widened slightly to include other open access publications.

      When I was contacted for comment, Bohannon seemed quite surprised by the suggestion that he might have obtained similar results from subscription-based journals. This flaw in experimental design is of course compounded by the assumption that the problem is attributable to open access, rather than peer review (Eisen’s analogy to Nigerian financial scammers is apt, except that in this case it is a Nigerian scientist who was caught in the scam). But I suspect that this was an honest mistake … although one can certainly see how it might heighten the appeal of the article to Science.

      It is ironic to consider that Bohannon’s mistake might have been caught by rigorous peer review. Indeed, even without a copy of the article to read, I raised this concern during our phone conversation, and my comment was highlighted in the published report. As noted in many of the responses to this post, isn’t it best to allow controversial work to be published … and then subjected to post-publication discussions like this one?

      Feel free to join the Google Hangout that Science is sponsoring on Thurs Oct 10:

  7. Gerry
    Posted October 3, 2013 at 1:18 pm | Permalink

    The fact that some of the journals got a peer review may not be entirely relevant. When I started my reviewing career, I reviewed 2 papers an open access journal. The first wasn’t terrible but there were substantial flaws. The second was absolutely awful with a one sentence long results section, poor methods and an overblown conclusion. I recommended rejection for both and both were published without edits shortly afterwards on the journal website. Sometimes the “peer review” is just for show.

    There is a form of QC for peer review – inclusion in the impact factor calculations and listing in Pubmed. I wouldn’t dream of publishing in a journal that did not have both of these.

  8. Posted October 3, 2013 at 1:27 pm | Permalink
  9. Posted October 3, 2013 at 3:29 pm | Permalink

    Since the name they gave to the arsenic miracle was GFAJ – Give Felise A Job, after the lead author – I wouldn’t have been surprised if it was a put-on. But I blame the NASA hype machine as much as anyone and if the co-authors from USGS, ASU, LLNL, Stanford and my alma mater, Duquesne, did not catch the problem, I am unsurprised corporate employees fell down on the job.

  10. Jim Woodgett
    Posted October 3, 2013 at 4:29 pm | Permalink

    Touché and well said Michael! Yes, there are a bunch of scammy open access journals that take advantage of ignorance but also serve those who wish to pay to publish material that wouldn’t pass muster in a decent OA journal (or closed access journal). We all know these low life journals exist but was it really necessary to expose them in this way? Who does it serve? The result is predictable and actually shows that peer review is effective in good journals. Who’s role is it to point out that certain companies make awful products? Moreover, this problem solves itself if people actually read the material published. There’s garbage published everywhere. Sometimes its lax review, other times its editors ignoring good peer review. But the result is typically the same. Crappy papers are recognized by other scientists and ignored.

  11. Jay
    Posted October 3, 2013 at 4:48 pm | Permalink

    “First, and foremost, we need to get past the antiquated idea that the singular act of publication – or publication in a particular journal – should signal for all eternity that a paper is valid, let alone important.”

    And yet day after day, I am inundated with appeals to authority based on a paper’s peer review.

    I wish you had gone a bit further and offered a simple metric for the media and public to use that can help determine if a paper is valid let alone important.

    Number of citations? I hear that bandied around a bit, but I’ve always wondered what happens to say the Arsenic DNA paper that presumably is cited a lot as a bad paper and then is given a high number of citations number….

    • Posted October 3, 2013 at 5:17 pm | Permalink

      I wish you had gone a bit further and offered a simple metric for the media and public to use that can help determine if a paper is valid let alone important.

      I think the problem here is that there just isn’t one. And if you ever did make one, it would only be a matter of time before people started gaming it–this is essentially Campbell’s Law.

      The best method seems to be transparency; make as much information as possible available and people will be able to sort the wheat from the chaff.

  12. Booker
    Posted October 3, 2013 at 5:46 pm | Permalink

    Great post, hilarious, but also gets right down to the crux of the matter.

  13. Posted October 3, 2013 at 11:10 pm | Permalink

    Entertaining piece.

    Of course, it gets worse. I can relate a story of an article I submitted to the premier physics letters journal (three guesses who). The editors dutifully obtained four reviewers who rejected it as “not sufficiently important”. I then exercised my right as author to have it considered for a less urgent member of their stable – where it was published.


    One year later someone from a very highly regarded institution published a paper on the same topic in the same journal using precisely the same original illustrative examples.

    Their was judged important enough to publish whereas mine was not.

    On examination, the mathematical scheme they employed was false. They had the right examples but the calculations were wrong!

    I published a letter to that effect which was published in that journal.

    Independently, somebody else pointed out the paper was wrong.

    Was it withdrawn? No.

    Is it cited? Hell, yes. Even though two separate authors pointed out it was wrong.

    This is where publication fails. There is tons of crap out there and some totally egregious and deeply flawed publication practises.

    I do not see a ready solution to this problem. I no longer publish. I keep my results to myself and I develop commercial applications of the back of them.

    The traditional publication system is deeply corrupt and thoroughly broken.

  14. Posted October 3, 2013 at 11:54 pm | Permalink

    Why do people use stings and anecdotes, when we already have data:

    ‘High-Impact’ journals are no better (in fact in some cases worse) than ‘lower’ journals. There is no need for a sting operation or other anecdotes to show that, we already have that data.

    • Andrea Campisano
      Posted October 10, 2013 at 12:20 pm | Permalink

      It’s a simple enough question, as it is the answer.
      We do because very often one single paper on those journals, how flawed they may be, means a permanent position or decent fundings.

      Seems to me a good enough reason.

  15. tartempion
    Posted October 4, 2013 at 4:37 am | Permalink

    I wonder, how many researchers were asked to review that paper more than once…

  16. Posted October 4, 2013 at 5:31 am | Permalink

    Once I had found a proof that Neuronal Nets need a negative feedback to achieve cognitive skills. By some experiments I could even show that biological neurons definitely had to have these negative feedbacks implemented. The paper was rejected.

    Four months later on a conference for complex systems someone presented his findings of how neurons can “perceive” an impulse they sent did not become a positive result ..

  17. Posted October 4, 2013 at 5:42 am | Permalink

    Peer review may not be a joke, but it is seriously broken. This excellent post highlights the problems with Science’s little journalistic foray into the lack of peer review at predatory OA journals (see Beall’s list of these http://scholarlyoa.com/publishers/). it also hits two other key points, the lack of rigour in Science’s effort and the lack of introspection on Science’s part with respect to their own practice. The same issues arise at NPG and many, many, other publishers. Editors generally do not act as editors and are incredibly reluctant to take editorial action.

    Peer review is better than no peer review.

    But, peer review by 2-5 individuals, no matter how talented they are is necessarily limited.
    Often, despite their exalted positions, these folk do not actually engage in any peer review. See the recent debacles over the chopstick nanorods paper and the comment from the PI to the postdoc on making up an elemental analysis that was left in the SI and published (see posts at http://blog.chembark.com/ ). Reviewers and editors clearly never read the SI, so the paper cannot be considered to have been peer reviewed.

    This happens a lot and concerns many of us; a reasonable number have commented on the problem at some length, including myself, e.g., http://ferniglab.wordpress.com/2013/08/15/does-science-self-right/

    I think the “solutions” have to include some sort of post publication peer review followed by robust editorial action, when it is required, again issues that many have engaged with, including myself, e.g., http://ferniglab.wordpress.com/2013/09/16/getting-science-right-side-up/

    One of these solutions is PubPeer.com, but we have to engage and really try to get editors to act. there are a fair number of papers on PubPeer where authors and journal are facing many serious critiques in stony silence.

  18. Adam Etkin (@adametk
    Posted October 4, 2013 at 6:19 am | Permalink

    Using the “Arsenic is Life” paper is too easy. We can’t ignore the fact that nearly 2,000,000 papers across 24,000 peer reviewed journals (and growing) are published each year. Of course mistakes will be made. Cherry picking one or two high profile cases to make your point is a cop out IMO. No one claims peer review is perfect. That does not mean that “peer review is a joke.” I’m surprised no one has made the point yet that one of the reasons that the “traditional” journals peer review process takes longer, something it is often criticized for, is due to the fact that a truly ethical, rigorous peer review process can take time. Would this “sting” have caught some traditional journals in it’s net? Probably. Would the large majority of those you relentlessly attack fallen for it? I’m confident they would not have.

  19. Posted October 4, 2013 at 6:44 am | Permalink

    This whole discussion is insane. I have plenty of papers in major journals, including Science and PLoS ONE, and, more importantly, plenty of REJECTED papers littering my backups. Yet, I seem to be the only scientist in the world who LOVES peer review! I treat peer review as data, and, as a good scientist, nothing is sweeter than data. Of course, I prefer good data, so I appreciate a good solid peer review. In fact, I worry when a journal takes one of my papers without complaining because I’m afraid that the peer review wasn’t deep enough. I’ve been spanked by peer reviewers more times than I can count, and I’m convinced that it’s only served to improve my science. In fact, with only one exception that I can recall (in ~100 published, and, maybe 25 rejected papers), every review that I have got, including the totally negative ones, have improved my papers; INCLUDING the ones that rejected my papers — every rejected paper of mine probably deserved it, and, in most cases, I would have been embarrassed to have published my papers without peer review (and, probably would have been really really embarrassed to have published the rejected ones!) I just don’t get why this discussion is going on at all; peer review is great free data, and more than that, it’s like having a free co-author that you don’t even have to share the credit with. What could be better than that?!

    • Posted October 4, 2013 at 7:42 am | Permalink

      The website field seems to have a length limit, which is silly — many website addresses as quite long. Anyway, as a result, the website pointer that I provided in the above post got truncated. Here’s what it was supposed to be:


    • Posted October 4, 2013 at 10:55 am | Permalink

      Re: Jeff
      It is true that peer review is valuable data. So why keep it secret and reduce it to a “yes” or “no” decision? Whys shouldn’t authors be able to acknowledge reviewers for their contributions and even invite them to collaborate? Isn’t this what science is about?

      • Jeff Shrager
        Posted October 4, 2013 at 2:10 pm | Permalink

        I feel that people should be free to comment/review either in public or private, and either anonymously or identifiably. That is, to my mind, the right of the reviewer. I have occassionally offered a reviewer an explicit acknowledgement, and have occassionally been taken up on the offer. (I have even once offered a reviewer to collaborate, but have not offered a reviewer a co-authorship merely on the basis of a review.)

    • Posted October 5, 2013 at 8:41 am | Permalink

      I love peer review when it is performed correctly. I do, however, loathe the widespread editorial practice of deciding to reject a paper without review due the editor not finding its abstract/title/author “significant” or sexy enough.

      • Anna O
        Posted October 6, 2013 at 4:46 am | Permalink

        I second that!

    • Aneesh Panoli
      Posted October 5, 2013 at 3:53 pm | Permalink

      Why still the cryptic act of “rogue reviewers” in this 21st centuary. make the entire review process public along with the paper.
      There was a time when your spouse could become a co-author without having contributed anything. The solution was adding author contributions section.

      Similarly, why not include reviewer questions and author comments? These days even internet doesn’t allow anonymity, why should there be anything at all in peer reviewing?

    • Posted November 4, 2013 at 8:58 pm | Permalink

      I agree with you. Proper peer review is ultimately to your benefit as an author, as it serves to protect your reputation, should you have made a mistake. Better a paper rejection and re-submission than the scenario of a paper accepted, found wanting, and retraction.

      In an a world where scholarly journal editors behave with integrity, and peer reviewers are impartial and fair, all would be well. However, something is amiss if the sort of incident described in this comment by K.J. occurs with even minimal frequency.

      It is my inclination to defend traditional scholarly publishing, Elsevier, peer review etc. Despite the fact that I want to “be on their side”, the contributors to Scholarly Kitchen are so unfriendly, arrogant, disdainful and arbitrary (not as publishers, but in dealing with their blog and comments, and not merely to me, but to very accomplished, credentialed, sometimes elderly individuals) that I shudder at the thought of contending with such personalities on a professional level. I find K.J.’s account quite convincing, plausible.

      Peer review should be as you describe! Whether OA or not-OA, cursory or biased peer review is useless or even harmful to all. How does one enforce responsible peer review? I don’t think that enforcement is possible. If there is a willingness to subvert the system, and that is true for researchers (who knowingly submit shoddy or unoriginal work) as well as peer reviewers, no metric will be sufficient to remedy improper attribution of research, low quality/ vanity publications and rentier behavior by journal management.

      Science has, and will have, credibility problems, if continuing on the same path. There are larger underlying issues, but this is too lengthy as is.

  20. Posted October 4, 2013 at 8:09 am | Permalink

    Thanks for writing this up! I have to say frankly, I was badly surprised by the following conclusion: “Bohannon doesn’t offer any real analysis of his sting operation, preferring apparently to let the results stand on their own with the clear implication that because of unscrupulous operators, open-source publishing is seriously flawed and in many cases should not be taken seriously by those wishing to publish research papers.” (http://phys.org/news/2013-10-paper-publishing-reveals-lax-standards.html)

    The quotation above clearly gives an example of flawed logic, but what is worse is that OA has been used as a cover for a new vawe of publishing scum.

  21. Colin Anderson
    Posted October 4, 2013 at 9:53 am | Permalink

    To Jeff Shrager: It is interesting to read an experience completely opposite to my own. I have rarely had a review that has had any impact whatsoever on the way we would write our paper or think about our subject. The rejections have almost universally appeared to be ‘political’ in nature; seemingly from the anonymous unscrupulous competitors. Michael Eisen’s evaluation is spot on in my experience. Anonymous pre-publication peer review is primarily a method for self appointed ‘elite’ scientists to maintain their position in the field. The cosy relationship between ‘elite’ journals and ‘elite’ scientists creates a self serving circle, benefiting those scientists (they are able to get weak papers in) and the bottom line of those journals. This circle could not happen without a substantial portion of scientists buying into the fallacy of the ‘elite’ scientist.

    • Jeff Shrager
      Posted October 4, 2013 at 2:04 pm | Permalink

      Wow. Our experience truly is completely the opposite. I can only speak from my own experience, but when I review, politics does not enter into it at all. (At least not consciously/explicitly. If we’re allowing a post modern interpretation of “politics” and “science” then I suppose politics always enters into it, even if I’m just making simple assertions of fact, but presumably we’re not going down that rat hole!) Moreover, I’ve only once in having been subjected to hundreds of reviews of my own work, experienced anything like what you are describing.

  22. Anne Carpenter
    Posted October 4, 2013 at 12:49 pm | Permalink

    Bohannon’s work here is a crime against the scientific community.

    How ethical is it for a reporter to waste the time of at least 100 journals x
    3 reviewers? I’m horrified. 300 people times a half day is 150 work days:
    that is 3/4 of a YEAR of scientific productivity the reporter stole from the
    scientific community, to prove a point everyone already knew: there are
    scammy journals out there.

    Science should be ashamed of itself for publishing this and thus
    affirming that it is okay for people to waste reviewers’ time on their own
    pet projects. And violating the submission standards of journals which
    require that manuscripts not be submitted to more than one journal at a time.

    Or is Science saying it is okay for the rest of us to ignore those rules? I’m not to eager to help Science out with reviewing their papers in the future, based on their callous disregard for scientists’ time. Shameful.

  23. Posted October 4, 2013 at 1:30 pm | Permalink

    The problem is that so many journals have a few editors handling tons of papers. we combatted this at HCB by having a lot of editors handling a few papers so that they get just prudence.

    Additionally, you mention how many papers are accepted that should not be. What about those that are rejected that should not be? Bad editor and peer review can work in both directions.


  24. Posted October 4, 2013 at 2:59 pm | Permalink

    Re: “…peer review is a joke….. This all adds up to showing that peer review simply doesn’t work.”

    Notwithstanding this blog entry and a previous blog entry by Dr. Eisen that he refers to, “Peer review is f***ed up – let’s fix it” (October 28, 2011), I believe that it is premature to conclude that peer review is “a joke…that simply doesn’t work.” I would seek more evidence to arrive at that conclusion. Disturbing and troubling the examples provided are, I hesitate to draw conclusions from them. A sad possibility is that peer review is the worst way to evaluate suitability for publication except for all of the others tried.

    One of these other ways to evaluate suitability for publication is to dispense entirely with peer review. Recalls the Sokal affair, in which physicist Alan Sokal submitted a nonsensical article to a social science journal. The journal, which did not practice peer review, accepted the article. For a description of this episode and others, see the Wikipedia entry, “Sokal affair” (http://en.wikipedia.org/wiki/Sokal_affair; accessed October 4, 2013).

    I wonder about opinions arrived at the International Congresses on Peer Review and Biomedical Publication, the seventh of which was held September 8-10, 2012 (www.peerreviewcongress.org).

  25. It's not obvious
    Posted October 4, 2013 at 6:00 pm | Permalink

    it’s been fascinating to see you and Suber get so hot about this paper. the vitriol you guys expel at “traditional” “paywall” “closed-access” publishing is really remarkable. I heard the author on NPR this morning he was much more level-headed than the OA folks have been in reaction to him. I find looking for the “heat” in such conflicts tells us a lot, and that where there is so much “”heat” there is often work that needs to be done at a very foundational level about what is motivating the conflict.

    in my opinion neither you nor Suber (or anyone else I’ve read, and it’s surprising given that Suber’s been around this long and has written a whole book about it) have yet to explain, in clear, principled, direct terms, what the social benefit is that you are pursuing. not vague terms like “freeing science” and not in specific critiques of existing abusive issues like (some) skyrocketing journal costs. You guys speak as if open access *publishing*–put aside repositories for the moment–is some kind of absolute moral good. The only way I can read that is to say that existing academic publication is a moral evil. If I agree with you that high journal costs are a problem, that doesn’t get me very far down the road toward your solution: I might more easily recommend a governmental or educational prohibition on journals exceeding a certain cost threshold.

    What is evil about Science and Nature? Seriously, in very specific terms. Not just that they are expensive, because if being expensive is a sin, then you know very well that there is a lot in the world of science and technology that is much more expensive than journal publication.

    You are on a moral crusade that I don’t understand.

    One of its effects, as this paper actually shows, is to deeply muddy the waters about what is and is not reliable academic publication. I’m not saying they weren’t muddy before; but they are much muddier now. Is that a benefit? Are we better off today than 10 years ago, when young scientists are much more unclear about where publications will have impact, help them get tenure, and be widely read and respected?

    Would it really be good if Science and Nature went away and PLoS One became the last man standing? What ethical or moral victory would that accomplish? Right now, I have been told by quite a few scientist acquaintances that PLoS One publication is a “joke.” and that’s one of the best of them. Publishing there makes tenure judgments harder, evaluation harder, and much else. and WHY? for what cause?

    I have read your work carefully and Suber’s work. I find you continually evasive when we get down to brass tacks. What is so awful about the current system and what is so wonderful about the system you propose to replace it, and how do you know?

    Further, given the fact that money is not going away anytime soon, the fact that you are essentially attacking the academic revenue stream, while leaving untouched the revenue stream of the big money people in our society, corporations, makes the moral crusade really hard to understand. You know who will really appreciate paying nothing for Science and Nature? Cisco, Proctor & Gamble, etc.

    I am asking you in all earnestness: explain in detail, in depth, and with reference to founding principles, what is the evil in the current system, and what is the good your system will create, and why you are so passionate about it, which seems to me all out of balance with the import of the issues. You write as if these are obvious, and I have to tell you, as an academic not in the sciences, the more I watch this and the more I look at it, I feel more and more that the opposite is the case, and that the palpable disdain you, Suber, and other hard OA advocates express for the “existing system”–without explaining exactly what is so wrong with it, how you know, and what it is you’d like to maintain about it, as well as what needs changing, with reference to clearly-articulated principles of the function and purpose of the academy–reads to me like a hatred of the academic project itself, a hatred that happens to tie very nicely with those corporate fatcats who can’t wait for we “rich” academics to make it that much easier for them to get their hands on our stuff.

    • Posted October 4, 2013 at 10:37 pm | Permalink

      You really must not read what we write very carefully, because the primary problem we are trying to address is that the the subscription-based publishing system denies the vast majority of people on Earth, including many scientists, students, teachers and physicians, as well as interested lay-people, access to the scientific literature. Full stop.

      This explains my point of view and reasons for feeling this way as well as anything:


      • Posted October 4, 2013 at 11:46 pm | Permalink

        Don’t get me wrong – I would prefer for all scientific journals to be run as non-profits.

        But this argumentation has never entirely convinced me. First, everybody who wants to read my papers can send me an eMail and request the PDF. That is how we roll. And if my institution does not have access to journal X, I send an eMail to a friend at a different institution who has. So far the only problems I have encountered were with really obscure, old journals.

        Second, it will always cost money. If it is not paid by the reader, it has to be paid by the author, and I am concerned about the effect that will have on the access of researchers from poorer countries or less well funded institutions to journals. Yes, some journals offer to waive or reduce fees, but if OA became the dominant model, would that still remain feasible?

        • Posted October 5, 2013 at 12:05 am | Permalink

          The costs do not need to be paid by the reader or the author – indeed they are rarely paid by either today. The overwhelming majority of the $10b/year spent on science and medical journals today (something like 70%) comes from funding agencies, universities and other research organizations. Everyone agrees that these same organizations that fund science should continue to support the dissemination of science. The question at hand is how they should do it. Should they spend $10b to buy access to the literature to a narrow slice of the planet’s population though subscriptions, or should they spend less (almost everyone agrees open access will cost less) to buy access for everyone on the planet.

          And the email request argument is absurd. If everyone could get copies of PDFs by emailing the authors, nobody would subscribe to journals.

          • Posted October 5, 2013 at 1:53 am | Permalink

            Well, it doesn’t work for older literature obviously, especially if the author is deceased, but I regularly get requests for PDFs from colleagues especially in developing countries. Is it different in your area?

            I am not sure how OA could be assumed to cost less qua OA; if it does, it would do so because it would be run by non-profits instead of Elsevier et al. Otherwise, where are the potential savings coming from? And if that is the difference, why not run a non-profit subscription based model?

            Really it boils down to two trade-offs: On the plus side, OA makes results available to everybody, but on the downside, it shifts incentives from accepting quality to accepting quantity of papers because the journals are paid per paper accepted and not per issue bought.

            On the plus side, poor libraries get access, but on the downside, poorly funded scientists may be barred from publishing.

            I am not entirely convinced that the OA side of these trade-offs is as great as some envision. What do you do as a Bolivian scientist whose national institutions do not have the money to pay the publication fees for the good journals full stop? At the moment, you do not have access to the literature but you can ask a colleague in the USA or Germany to get the PDF for you. In OA utopia, you cannot publish. What is better?

          • Posted October 5, 2013 at 7:48 am | Permalink

            I actually favor a system in which funding agencies directly subsidize journals and there are no subscriptions or author charges. It’s the most efficient was to do it.

            In either case, it is a negligible cost to the system to subsidize publication from labs (including those in wealthy countries) who have limited funding. Also, the costs are coming down rapidly – look at PeerJ, for example.

          • Posted October 5, 2013 at 5:49 pm | Permalink

            Agreed, it would be best if both ends of the pipeline were free. Here in Australia there is a peer reviewed taxonomic journal that now works like that, run by the herbarium in Perth: http://florabase.dpaw.wa.gov.au/nuytsia/

            Its model is fantastic (no IF though). But it means that somebody has to be willing to fund the costs that still remain.

        • Posted October 5, 2013 at 8:51 am | Permalink

          ” At the moment, you do not have access to the literature but you can ask a colleague in the USA or Germany to get the PDF for you. In OA utopia, you cannot publish. ”
          That is not true: the case in point is ArXiv. Submission is free, reading is free, operating costs are negligible. As a pre-print server, there is no formal peer review, but that has not prevented it from being the most significant repository of physics/mathematic research.
          The only problem preventing us from moving everything to an ArXiv-like model is sociological: evaluation/promotion commitees use journal name recognition (i.e. “brand”) as a proxy to the quality of the evaluated work, instead of reading it themselves.

          • Gavin
            Posted October 5, 2013 at 2:00 pm | Permalink

            ArXiv is not free to run – current expenses are ~$800,000 per year. Expanding it to all fields would need a completely new business model.

            More importantly you are ignoring the absolute need for ‘short-cuts’ in evaluating what literature (out of thousands of papers published each month) to pay attention to. Given this need, there will always be a role for aggregators and clues as to interest and/or quality. Right now that role is (imperfectly) played by big ‘high impact’ journals and people submit to match. Who plays this role in the envisaged #OA future?

          • Posted October 5, 2013 at 5:58 pm | Permalink

            What Gavin said: If everything were just dumped onto an archive, how do I know whether it is good? How will a complete layperson tell the nonsense from the decent science?

            Yes, peer review is faulty because it is done by humans but it performs at least some gate keeping function. If something new is published in the journal Systematic Biology in Evolution I can afford to have more trust in it than if it is just a manuscript PDF somebody plopped down somewhere.

            How would QC work in an archive? Many people seem to think that we should then have some kind of likes or up/down-votes system, or they envision peer review being done in comment streams under the paper. But it is clear that the likes or votes are terribly easily gamed, and as for the second, where would a severely time constrained academic get the incentive to do something about the criticism expressed AFTER the publication of their paper on the archive? It is published, I can cite it and put it into my CV, time is wasting, on to the next project!

            No, it is good to have a review process before publication to get rid of the obvious trash, even if a bit of trash still sometimes comes through the gate.

  26. Brad
    Posted October 5, 2013 at 6:40 am | Permalink

    Have you seen the IPCC AR5 report? World Wildlife Fund and Greenpeace cited instead of peer review, neglecting to mention science that goes against their argument, and spinning what they do cite. Truly awful science but academia signs off as it fits their political view. Amazingly bad peer review.

    Try it, look at the real data: climate4you.com

24 Trackbacks