PubMed Commons: Post publication peer review goes mainstream

I have written a lot about how I think the biggest problem in science communication today is the disproportionate value we place on where papers are published when assessing the validity and import of a work of science, and the contribution of its authors. And I have argued that the best way to change this is to develop a robust system of post publication peer review (PPPR) , in which works are assessed continuously after they are published so that flaws can be identified and corrected and so that the most credit is reserved for works that withstand the test of time.

There have been LOTS of efforts to get post-publication peer review off the ground – usually in the form of comments on a journal’s website – but these have, with few exceptions, failed to generate sustained use. There are lots of possible reasons for this – from poor implementation, to lack of interest on the part of potential discussants. However, I’ve always felt the biggest flaw was that these were on journal websites – that you had to think about where the work was published, and whether they had a commenting system, and whether you had an account, etc…

What we’ve always needed was a central place where you know you can always go to record comments on a paper you are reading, and, conversely, where you can get all of the comments other scientists have on a paper you’re reading or are interested in. There have been a couple of services that have tried to create such a system – cf PubPeer, which lets you comment on any paper in PubMed – but they have been slow to gain traction in the community.

The obvious place to build such a commenting/post publication review system has always been directly in PubMed – it has everything and everyone already uses it. This is why I am excited – and cautiously optimistic – about a new project called PubMed Commons that will allow registered users (for now primarily NIH grantees) to post comments on any paper in PubMed, which will then appear alongside the paper when it is received in a search.

Here is how PubMed Commons describes itself:

PubMed Commons is a system that enables researchers to share their opinions about scientific publications. Researchers can comment on any publication indexed by PubMed, and read the comments of others.

PubMed Commons is a forum for open and constructive criticism and discussion of scientific issues. It will thrive with high quality interchange from the scientific community.

The system is still pretty threadbare – it only allows simply commenting, and not, for example, rating of the work – but I’ve used it and it is easy to get in, comment and get out. A lot more info on the project can be found here.

This is a great opportunity for us to make PPPR real. But it’s only going to work if people participate. So, if you’re an NIH grantee, and you want to see science communication improve, make a commitment to comment in a paper you’ve read at least once a week, and let’s make this thing work!!

This entry was posted in open access, public access, publishing. Bookmark the permalink. Both comments and trackbacks are currently closed.

14 Comments

  1. Posted October 22, 2013 at 8:13 am | Permalink

    Very nice, but the ludicrous arbitrary restriction on who can participate makes this completely irrelevant to me, and to most of my colleagues. Unfortunately, it makes this look like empire-building rather than community-building.

    Let me know if they ever make this open-science initiative, y’know, open.

    • Posted October 22, 2013 at 12:29 pm | Permalink

      I don’t think the restriction is so ludicrous. Open online comments are notorious for attracting vocal wackos. PubMed simply doesn’t have the resources to put into moderating an open forum, so they had to pick some criterion to get the ball rolling with.

      In the early stages I think it will be more important to establish legitimacy and usefulness. I imagine once that’s proven PubMed will look at how best to make sure all qualified voices are included.

    • Posted October 22, 2013 at 12:42 pm | Permalink

      The site is open to anyone who has published a paper in PubMed. Doesn’t seem that ludicrous to have the community of authors commenting on papers. I would love to see it more open, but it’s a not unreasonable restriction for the launch. The system for getting invited is a bit byzantine, but I think that will change.

      • Posted October 22, 2013 at 1:09 pm | Permalink

        I have papers in PubMed, but can’t comment without a special dispensation from a most-favoured author. I don’t call that any kind of open.

        • Posted October 22, 2013 at 1:41 pm | Permalink

          That’s what the NIH required to allow it to go forward at the start. Yes, it’s a bit annoying. But I’ll invite anyone who asks.

        • Garry Myers
          Posted October 22, 2013 at 1:49 pm | Permalink

          From the website FAQ (http://www.ncbi.nlm.nih.gov/pubmedcommons/faq/) – “PubMed Commons is currently in a closed pilot testing phase, which means that only invited participants can add and view comments in PubMed”

          A closed pilot is by definition..not open. Seems overly harsh to be criticizing a restricted pilot project for not being fully open from the outset. It’s pretty basic at the moment but the bones are there for this to be very useful. I’d be very cautious about dismissing an overtly “beta” tool.

          • Posted October 23, 2013 at 4:04 am | Permalink

            OK, that’s all fair enough. They can do what they want in a “closed pilot” stage. I’ll be joining in when they reach the open stage.

  2. David S
    Posted October 22, 2013 at 1:12 pm | Permalink

    Already my experience is that the moderators are excessively intrusive and the comments are not going to be very informative about the limitations of the published research.

  3. GavinS
    Posted October 23, 2013 at 3:32 am | Permalink

    The obvious place to build such a commenting/post publication review system has always been directly in PubMed – it has everything and everyone already uses it.

    PubMed does not index everything, let alone everything in science. Journals are vetted and can be rejected for inclusion. And what is the evidence that ‘everyone’ uses it? These assertions remind me of the cartoon of the drunk looking for his keys under the lamp-post, because that is where the light is.

    • Posted October 23, 2013 at 7:05 am | Permalink

      Yes. I should have said it is the most widely used search engine for the biomedical sciences, and it has a far more comprehensive collection than does any publisher.

      • Posted October 23, 2013 at 1:39 pm | Permalink

        Except for PubPeer, which has every article ever published (and allows comments to be made anonymously).

  4. Posted October 23, 2013 at 5:17 pm | Permalink

    The Selected Papers Network is a version of this that works for arXiv papers, papers in PubMed and anything with a DOI

    https://selectedpapers.net/

    At present one needs a Google account to use the site, which is probably not as open as Mike Taylor would like, but the eventual aim is to be able to farm comments from social media including blogs and twitter via curated hash tags (at present it only works using Google+, but that’s only because lots of mathematicians are on G+). See http://docs.selectedpapers.net/intro.html for more explanation.

  5. top1214
    Posted November 27, 2013 at 10:37 pm | Permalink

    That’s really interesting. I wonder if emailing the corresponding author(s) when a comment is made on one of their submissions will be even better than emailing the author directly. Not that I know if this will be a feature, but I would like it to be.

  6. Andrey Chursov
    Posted December 15, 2013 at 3:11 pm | Permalink

    There is another system aimed to improve science communication:
    http://www.ScientificCitations.org
    It is a system that enables researchers to discuss and add missing references to already published scientific papers. Such Post-Publication Citations are relevant references that authors agree to add to their previously published list of referenced literature and which are freely available on the website.

3 Trackbacks