I think the necessity for and credibility of peer review is an interesting issue. On the one hand, it is a system of checks and balances to ensure that authors meet the accepted standards of their discipline and reduces the dissemination of irrelevant findings, unwarranted claims, unacceptable interpretations, and personal views. An article having gone through peer review automatically affords it a certain degree of credibility in the eyes of readers and fellow academics. Publications that have not undergone peer review are likely to be regarded with suspicion by scholars and professionals.
Whether this level faith or trust in the system is merited can be argued; Peer review may make the ability to publish susceptible to control by elites and potentially, even personal jealousy and other kinds of bias. Reviewers tend to be especially critical of conclusions that contradict their own views, and lenient towards those that accord with them. Also ideas that harmonize with the established experts' (generally who gets chose n to be a peer reviewer) in a field are more likely to see print and to appear in premier journals than are iconoclastic or revolutionary ones [1].
For the purposes of this blog post and keeping with the controversy theme (the Sokal affair that Prof. Galey brought up), I thought it might be interesting to look up some incidents of 'Peer Review failure'. Failure, in this context, is defined to be publishing any kind of work containing obvious fundamental errors that undermines at least one of its main conclusions, when a journal publishes well-known information as a new discovery, or when important valid work is rejected out of hand. Retractions and letters-to-the-editor that correct major errors in articles would generally constitute peer review failures [2].
The Wikipedia article on Peer Review failure put me on to the publication of 'Tai's method' [3] for calculating areas under curves (in this case glucose tolerance and other metabolic curves) as original research. The method of Riemann sums for numerical integration, which was republished, in this particular paper is a technique taught in high school calculus and something that I've used personally a fair. So naturally I found that particularly intriguing and even amusing. And apparently it is a prominent example of a well known idea being re-branded as a new discovery. I learnt that Edward Jenner's report of the first vaccination against smallpox was rejected too!
I guess we have to concede that peer review, in scientific journals, assumes that the article reviewed has been honestly written, and the process is not designed to detect fraud. The reviewers usually do not have full access to the data from which the paper has been written and some elements have to be taken on trust. Therefore peer review is not considered a failure in cases of deliberate fraud by authors. It is not usually practical for the reviewer to reproduce the author's work, unless the paper deals with purely theoretical problems which the reviewer can follow in a step-by-step manner.
I think the system of open peer review and leveraging the power of the internet to obtain rapid and detailed feedback is fascinating. Of course, filtering out the quality feedback from the rest due to the sheer volume of inputs is tricky but I'm certain that with advances in natural language processing and machine learning, such systems can be concerned as a seriously viable alternative. I looked up the Shakespeare quarterly experiment and thought it was a great first step towards such a movement [4].
I also think that disclosing the identities of reviews to the authors should ideally facilitate healthy discussion between both parties that leads to better science. At the end of the day, it is people's biases and egos that stop this from happening and it is important for scholars to rise above this and look at bigger picture.
REFERENCES
[3] Tai, M. M. (1994). A mathematical model for the determination of total area under glucose tolerance and other metabolic curves. Diabetes care, 17(2), 152-154.
Whether this level faith or trust in the system is merited can be argued; Peer review may make the ability to publish susceptible to control by elites and potentially, even personal jealousy and other kinds of bias. Reviewers tend to be especially critical of conclusions that contradict their own views, and lenient towards those that accord with them. Also ideas that harmonize with the established experts' (generally who gets chose n to be a peer reviewer) in a field are more likely to see print and to appear in premier journals than are iconoclastic or revolutionary ones [1].
For the purposes of this blog post and keeping with the controversy theme (the Sokal affair that Prof. Galey brought up), I thought it might be interesting to look up some incidents of 'Peer Review failure'. Failure, in this context, is defined to be publishing any kind of work containing obvious fundamental errors that undermines at least one of its main conclusions, when a journal publishes well-known information as a new discovery, or when important valid work is rejected out of hand. Retractions and letters-to-the-editor that correct major errors in articles would generally constitute peer review failures [2].
The Wikipedia article on Peer Review failure put me on to the publication of 'Tai's method' [3] for calculating areas under curves (in this case glucose tolerance and other metabolic curves) as original research. The method of Riemann sums for numerical integration, which was republished, in this particular paper is a technique taught in high school calculus and something that I've used personally a fair. So naturally I found that particularly intriguing and even amusing. And apparently it is a prominent example of a well known idea being re-branded as a new discovery. I learnt that Edward Jenner's report of the first vaccination against smallpox was rejected too!
I guess we have to concede that peer review, in scientific journals, assumes that the article reviewed has been honestly written, and the process is not designed to detect fraud. The reviewers usually do not have full access to the data from which the paper has been written and some elements have to be taken on trust. Therefore peer review is not considered a failure in cases of deliberate fraud by authors. It is not usually practical for the reviewer to reproduce the author's work, unless the paper deals with purely theoretical problems which the reviewer can follow in a step-by-step manner.
I think the system of open peer review and leveraging the power of the internet to obtain rapid and detailed feedback is fascinating. Of course, filtering out the quality feedback from the rest due to the sheer volume of inputs is tricky but I'm certain that with advances in natural language processing and machine learning, such systems can be concerned as a seriously viable alternative. I looked up the Shakespeare quarterly experiment and thought it was a great first step towards such a movement [4].
I also think that disclosing the identities of reviews to the authors should ideally facilitate healthy discussion between both parties that leads to better science. At the end of the day, it is people's biases and egos that stop this from happening and it is important for scholars to rise above this and look at bigger picture.
REFERENCES
[1] Higgs, Robert (May 7, 2007). "Peer Review, Publication in Top Journals, Scientific Consensus, and So Forth". Independent Institute. Retrieved April 9, 2012.
[2] Wikipedia article: Peer Review Failure - http://en.wikipedia.org/wiki/Peer_review_failure
[3] Tai, M. M. (1994). A mathematical model for the determination of total area under glucose tolerance and other metabolic curves. Diabetes care, 17(2), 152-154.
[4] Cohen, Patricia (2010) - Scholars Test Web Alternative to Peer Review -
http://www.nytimes.com/2010/08/24/arts/24peer.html?pagewanted=1&_r=2&ref=arts
http://www.nytimes.com/2010/08/24/arts/24peer.html?pagewanted=1&_r=2&ref=arts
No comments:
Post a Comment