Science. Communication. Community.
A new online commenting system on PubMed, one of the leading databases of peer-reviewed biomedical research, has opened the door for scientists to provide honest feedback about the pros and cons of their peers’ research.
There’s a new place to leave comments on the internet, and I couldn’t be more excited. PubMed, the US government’s online database of peer-reviewed biomedical research, has begun testing a new system to leave feedback on any article in its index.
Ironically, I recently wrote about the removal of the commenting system on Popular Science’s website and how this was actually a positive move for the scientific community. I don’t think I’m being inconsistent when I say that PubMed, which has done exactly the opposite, is also doing the best possible thing for the scientific community.
PubMed, which is the go-to publication database for many researchers in the biomedical research community, has finally implemented a viable system of post-peer review. What does that mean? First, a quick summary of how publication works in the sciences.
When researchers make a discovery, they write a short manuscript of a few thousand words summarizing what they found, how they did it, and why it’s important. They then submit their manuscript to a journal, like Science or Nature. The editors at this journal send the draft out to two or three individuals who are ostensibly experts in the field to review it. Their task is to ensure that the paper’s methods are appropriate to assess the topic at hand, as well as to ensure that the manuscript is relevant and framed appropriately (e.g. no factual errors or hype). Their comments, which are frequently anonymous, are then returned to the authors.
If the reviewers do not recommend that the paper be rejected for publication, the authors are usually expected to run additional experiments and/or rewrite sections the reviewers found objectionable. This revised manuscript is then sent back to the reviewers for another round of private commenting, and the process repeats until the manuscript is either rejected due to fundamental flaws or accepted as ready for publication. At this point, we say that the document has been “peer-reviewed”, and it shows up in databases like PubMed, Google Scholar, or Web of Science.
Once peer reviewed, these findings are considered tentatively valid by the scientific community. Rarely, a manuscript will later be formally retracted if it is revealed to be erroneous, but most retractions only occur upon revelation of scientific misconduct such as fabricating results.
Unfortunately, just because a paper has passed the peer review process doesn’t mean the information it contains is actually correct. Although reviewers may be experts on the topic, they are still humans who make mistakes, just like the authors. But once a manuscript is in print, its conclusions are difficult to challenge, until now.
Enter, post-peer review.
In the new PubMed Commons, scientists can leave comments on peer-reviewed articles. Perhaps they disagree with some of the interpretation of the data in the paper, or perhaps they were inspired by it. Perhaps another peer-reviewed publication has a contradictory finding that needs to be reconciled, or perhaps a new publication confirms this finding using a different method.
One of the fundamental tenets of the scientific method is reproducibility. A single study cannot establish that a finding is fact or fiction; instead, a family of independent studies conducted by different individuals with different instruments at different institutions must be considered together to build a body of evidence supporting a scientific conclusion. Unfortunately, as a result of the phenomenon of publication bias (most peer-reviewed publications only report studies that succeed, not ones that fail), word is unlikely to spread when scientists fail to reproduce published research claims. If evidence supporting a scientific conclusion is published in the public domain but evidence to the contrary isn’t, readers will have little reason to challenge the published finding. Post-peer review commenting systems will provide an easy method to get the word out when a publication doesn’t stand up to the rigor of reproducibility.
One of the main problems plaguing internet commenting systems is the blight of low-quality posts (off-topic posting, misinformation, name-calling, or the dreaded “trolling”). Currently, PubMed Commons is in a limited pilot phase where only a select group of scientists with specific credentials at particular institutions are allowed to post or view comments (disclosure: I’m a member of that group). Much like the feedback on sites like Amazon, other readers can give individual comments a thumbs up or down to denote helpfulness.
For now, the comments seem very high quality, particularly because they are being actively policed by the PubMed Commons staff. Moderators are quick to remove any post that does not appear to be positively contributing to the conversation (a relatively easy task in a limited-membership trial but perhaps a more difficult one if the commenting system is opened to the general public). My favorite comment that I’ve seen so far was a note from the original authors of a paper noting that their data, which was freely available online, had moved to a new Internet address. Because it’s effectively impossible to modify a peer-reviewed publication after it’s gone to press, even in the internet age, this individual was using the commenting system to ensure others could confirm his findings, upholding this pillar of the scientific method.
In order to keep commenters honest, individuals are required to sign their posts with their full name. This certainly helps minimize defamatory comments, but it also may make individuals think twice before they post something valid but negative. The peer-review system of publication (and grant applications) requires submissions to be evaluated by outside experts in the field. Some may not wish to publically comment in PubMed Commons for fear of reprisal from their peers. Particularly at early stages in one’s scientific career, a single grant application could be the difference between tenure and finding a new job. Irritating your potential reviewers could be a risky move.
At this stage in its trial, PubMed Commons seems to be a success. As of this writing, only 226 articles (out of about 23 million indexed on PubMed) have comments on them, but the pilot has only been running for a little over a week to a limited membership. In time, the comment base will expand to share important and relevant discourse on research in the biomedical sciences. Although the commenting system on Popular Science‘s website may have inhibited positive interpretation of science, the comments on PubMed Commons may have the potential to usher in a new era of scientific peer review.