PDA

View Full Version : Integrity of peer-review process



Igor_Goldenberg
28-11-2009, 09:54 PM
ClimateGate article (http://www.theaustralian.com.au/news/opinion/cool-heads-needed-to-turn-climate-change-heat-into-light/story-e6frg6zo-1225804731779) in The Australian touches a very serious problem:


Nevertheless, peer reviewing worked for many years as a more or less adequate system of quality control. In the end, the damage caused by cliquishness tended to be overcome through debate and the triumph of scientific integrity. But the situation has changed. Unfortunately, in some disciplines peer reviewing has become politicised. The way peer review is now used in public debate as a form of divine revelation - where we are told peer-reviewed science shows we must believe and do certain things - indicates how this institution risks being corrupted by advocacy researchers.

Spiny Norman
29-11-2009, 05:28 AM
I am really glad this is being discussed publicly now in the mainstream media. Peer review has unfortunately become something skin to "the prophets spake thusly" and we are all expected to fall into line. Its Scientism, not Science. It has become both a political and a philosophical tool that is used by some (not many, not all, just some) to prop up faltering belief systems.

Rincewind
29-11-2009, 10:06 AM
I am really glad this is being discussed publicly now in the mainstream media. Peer review has unfortunately become something skin to "the prophets spake thusly" and we are all expected to fall into line. Its Scientism, not Science. It has become both a political and a philosophical tool that is used by some (not many, not all, just some) to prop up faltering belief systems.

I think you are going just a bit too far there. There has never been any claim that peer-review was flawless and always works. Indeed there have been a number of cases where peer-review has failed in the past and will again in the future. However, it is a vital part of the operation of science and sorts out the chaff from the wheat.

In particular if a scientific paper is written without all the facts (concealing unfavourable results) or with manipulation of the data, that is something which peer-review has a very low chance of detecting. Fortunately what will find out these results is other independent groups verify the results post-publication.

A famous case where this happened was that of the (ex-)physicist Jan Hendrick Schön (his doctoral degree was revoked due to misconduct) who got away with making up results for a number of years. In the end, something like 20 papers had to be withdrawn by Science, Nature and Physical Review.

The thing that concerns me about Climate Science is that I have the perception that only a (relatively) small number of people are working in the field and a reasonably large portion of the research is advocacy funded. If true, that is a recipe for politicisation of the discipline.

PS From the link in my sigline...


Does peer review detect fraud and misconduct?

Peer review is not a fraud detection system. Referees are likely to detect some wrongdoing, such as copying someone else’s research or misrepresenting data, because they care about their subject. They know what research has been conducted already and the kinds of results that are likely. However, if someone deliberately sets out to falsify data, there is sometimes no way of knowing this until the paper is published and others in the scientific community scrutinise and try to repeat the work.

Kevin Bonham
29-11-2009, 03:04 PM
Unfortunately in the climate-emails case what is being discussed is not fabrication of data by researchers, but political interference in the refereeing process. This sort of interference can occur in various ways. Journal editors can deliberately send papers to hostile reviewers who will recommend that the papers be rejected, or biased reviewers can reject papers for bogus reasons.

Some years ago one of my colleagues attempted to have a paper published concerning the spread of bumblebees in Tasmania (bumblebees were introduced here in 1992 and are now abundant). A seemingly harmless subject but the debate is actually contaminated by vested interest because bumblebees are excellent for tomato pollination and therefore some tomato growers would like them to be widely introduced. A major barrier to this is the counter-argument that bumblebees will go feral and interact adversely with native insects. So for a while there was a denialist line that bumblebees were adapted to European vegetation and would not forage in native forests, which as it turned out was bulldust.

The upshot of all this is that when my colleague submitted his paper backed by records taken from all over the state by dozens of scientists (including me) the paper was rejected because one of the referees had stated that in his view the paper's results were not credible and had been fabricated. The journal got cold feet and declined to publish.

This sort of political interference was defeated by submitting the same draft to another journal but with all the records contributors who were willing to have their names thus included listed as authors. Thus, instead of one author the paper now had dozens and no editor would take seriously that all these people would be engaged in bumblebee-data fraud.

Typically the competition for the prestige of publishing important results first takes care of this sort of nonsense sooner or later.

Rincewind
29-11-2009, 05:22 PM
Unfortunately in the climate-emails case what is being discussed is not fabrication of data by researchers, but political interference in the refereeing process.

Sounds like a bit of both to me.


Journal editors can deliberately send papers to hostile reviewers who will recommend that the papers be rejected, or biased reviewers can reject papers for bogus reasons.

Yes all of that does happen but unless there is a world-wide conspiracy, reasonable work should be publishable somewhere.

Regarding the bumblebee study, was it submitted elsewhere?

Kevin Bonham
29-11-2009, 05:39 PM
Regarding the bumblebee study, was it submitted elsewhere?

Yes, and published.

Igor_Goldenberg
29-11-2009, 07:10 PM
Unfortunately in the climate-emails case what is being discussed is not fabrication of data by researchers, but political interference in the refereeing process. This sort of interference can occur in various ways. Journal editors can deliberately send papers to hostile reviewers who will recommend that the papers be rejected, or biased reviewers can reject papers for bogus reasons.

This is bound to happen in any area that became ideological. The further particular discipline from science the easier it's achieved.

TheJoker
30-11-2009, 03:13 PM
This is bound to happen in any area that became ideological. The further particular discipline from science the easier it's achieved.

IMO Climate Change science has more in common with economics and market analysis than typical science. Being that all the findings are essentially based on regression analysis of historic data to predict future trends. There is no possibility to conduct a controlled scientific experiment to verify or test hypothesis/theory.

Igor_Goldenberg
30-11-2009, 03:21 PM
IMO Climate Change science has more in common with economics and market analysis than typical science. Being that all the findings are essentially based on regression analysis of historic data to predict future trends. There is no possibility to conduct a controlled scientific experiment to verify or test hypothesis/theory.
That problem is magnified by two factors:
1. Unreliability of historic data pre-dating precise measurements
2. Lack of knowledge of statistics generally and regression analysis particularly. As a result they often don't know how to validate anything but a simple model.

Igor_Goldenberg
04-12-2009, 11:20 AM
Gloabl warming alarmist site published an interesting article about peer-review being necessary but not sufficient (http://www.realclimate.org/index.php/archives/2005/01/peer-review-a-necessary-but-not-sufficient-condition/). They picked one problem - that bogus paper can be published with peer-review mechanism (no wonder they try to illustrate it with climate-sceptics articles).
They missed the other - genuinely good high quality paper can be rejected for the same reasons - editor and/or reviewer have agendas the article does not support.

Kevin Bonham
04-12-2009, 12:13 PM
Peer review is basically a glorified form of argument-from-authority. It is usually the case that the peer-review process eliminates junk, but not necessarily. It's also possible for valid findings to be unfairly blocked in peer review although as Rincewind points out this form of obstruction tends not to last very long before the papers are published elsewhere, if the conclusions are basically valid.

I have seen far too much crap in my own field get through peer review. Indeed in my own field there is no real difference in quality between a lot of the "peer-reviewed" stuff and the so-called "grey lit" (consultants' reports, theses and so on). In my own case there is a heap of stuff I have done that is just not worth the effort of writing up for peer review unless someone wants to pay me to do it, but that doesn't make it any better or worse than the stuff in the 20+ peer-reviewed papers I have published.

I was looking at the International Code of Zoological Nomenclature recently to examine the requirements for describing a species name so that it is validly available to other taxonomists. Intriguingly, although there are requirements that a description must be issued in a form consisting of a number of copies produced simultaneously and available to the public either for free or at cost, there does not seem to be anything there that says a species name must be peer-reviewed. In theory one can self-publish; the question then is whether other scientists will subjectively decide to accept and use the names in question or not.

MichaelBaron
04-12-2009, 12:31 PM
My academic experience makes me smile when i think about ''validity'' of peer-reviews.

Firstly, even if review is carried out without revealing the authors' names it is usually easy to work out the authors' names based on the list of references used and reviewers knowledge of the subject matter experts.
Secondly, Some people are ''expected'' to get a good review even if the paper is written primarily by their co-authors (in other words, their students write for them :owned: ) and their name serves as a shield to protect the paper from potential attacks.

Thirdly, an academic review process is regarded as valid if the paper is sent to at least 2 different referees. Should only 1 of the reviews be negative, it does not stop the publisher/conference organizer from accepting the paper.

Finally, a paper is reviewed for a publication....there is always a possibility of rejection/paper being returned to the authors with suggestions for further improvements but conference papers rarely get rejected as conference organizers want as many people as possible to attend :).

And as a bonus point, I would like to note that academics/subject matter experts usually either work together or go to another extreme due to cut-throat competition. Consequently, they hate/love each other religiously and objectivity of peer reviews can hardly be regarded as realistic to achieve.

Rincewind
05-12-2009, 04:40 PM
My academic experience makes me smile when i think about ''validity'' of peer-reviews.

Out of interest, in which discipline did you have these experiences?

MichaelBaron
06-12-2009, 01:40 PM
Out of interest, in which discipline did you have these experiences?

Business, IT and communications

Vlad
06-12-2009, 02:37 PM
IMO Climate Change science has more in common with economics and market analysis than typical science. Being that all the findings are essentially based on regression analysis of historic data to predict future trends. There is no possibility to conduct a controlled scientific experiment to verify or test hypothesis/theory.

Very good point. The huge difference between Math and Economics (I am not even talking about its derivatives: Finance, Bussiness, etc.) is that in the majourity of Math journals the only what the referees and editors are concerned about is whether the results are correct. In Economics they also are concerned if the results are interesting, which makes everything very subjective. Imagine you are receiving your paper back with "Yeah, it looks right but the majourity of the readers will not find it interesting".

The quatative result is that the acceptance rate for most journals in math is not less than 50%. In Economics the acceptance rate for a good journal is 10%, for a top journal it is less than 5 %.

Rincewind
06-12-2009, 03:04 PM
Imagine you are receiving your paper back with "Yeah, it looks right but the majourity of the readers will not find it interesting".

That can happen in maths (or any discipline) if you submit it to the wrong journal. In applied maths it can happen if you submit it to a science journal because you think the results will be interesting to the audience but sometimes the audience just say "no". :)

What you say is even true applied maths journals which may reject a paper that is mathematically correct if it isn't sufficiently motivated. The answer will be "Yeah looks right but publish it in a pure maths journal".


The quatative result is that the acceptance rate for most journals in math is not less than 50%. In Economics the acceptance rate for a good journal is 10%, for a top journal it is less than 5 %.

I heard there was a study which showed that interdisciplinary variation in rejection rates had more to do with consensus than anything else.

Edit: I also suspect that top maths journals acceptance rates are less than 50%. I remember reading somewhere that Proc. Roy Soc Series A is around 20%.

Additional Edit: Interestingly, I read in a recent (2009) editorial the Sir Michael Berry reports that the rejection rate has been brought down from 80%+ to 72% which he feels is more appropriate. http://rspa.royalsocietypublishing.org/content/early/2009/11/07/rspa.2009.0535.full

Vlad
06-12-2009, 04:39 PM
That can happen in maths (or any discipline) if you submit it to the wrong journal. In applied maths it can happen if you submit it to a science journal because you think the results will be interesting to the audience but sometimes the audience just say "no". :)

What you say is even true applied maths journals which may reject a paper that is mathematically correct if it isn't sufficiently motivated. The answer will be "Yeah looks right but publish it in a pure maths journal".


I was talking about a well motivated paper which was sent to the right journal.
In Economics it is very common to get a paper rejected even though the result is right, the topic is right for the journal, the paper is well-motivated. The referees can just say that it is not interesting. They can also say that it contradicts their intuition which means it must be wrong.

The full process is very subjective. That leads to the situation when there are groups of scientists united in coalitions that support each other and reject papers of others. I think this is what the author of the article was talking about and I can confirm that it is quite common in Economics.





I heard there was a study which showed that interdisciplinary variation in rejection rates had more to do with consensus than anything else.



I personally would not think this result makes any sense. It is all about how much value you get from a publication. If you publish just one paper in one of the top Economics journals you are pretty much guaranteed to become a full prof in a few years in one of the top 8 Australian Universities, that is a permanent job with a salary 150+. Yes, you will probably need more than 1 paper altogether, but other papers will not be that difficult to publish. The real problem is to publish that top five paper and only a few people can do that and that is why it is only 5% acceptance rate.

Vlad
06-12-2009, 04:45 PM
When I was at ANU one of my profs told me the following story.
He submitted a paper to a journal and got it rejected. He received two reports. One was saying that the paper is trivial and this is why it needs to be rejcted. The other report was saying that the paper is wrong. He was trying to argue with the editor that it is impossible. It is either trivial and correct or non-trivial and wrong. It can't be both trivial and wrong. Anyway, that did not help, the paper was still rejected.:)

Rincewind
06-12-2009, 04:52 PM
When I was at ANU one of my profs told me the following story.
He submitted a paper to a journal and got it rejected. He received two reports. One was saying that the paper is trivial and this is why it needs to be rejcted. The other report was saying that the paper is wrong. He was trying to argue with the editor that it is impossible. It is either trivial and correct or non-trivial and wrong. It can't be both trivial and wrong. Anyway, that did not help, the paper was still rejected.:)

Maybe both are simultaneously possible in Economics. :D

MichaelBaron
08-12-2009, 01:01 PM
I was talking about a well motivated paper which was sent to the right journal.
In Economics it is very common to get a paper rejected even though the result is right, the topic is right for the journal, the paper is well-motivated. The referees can just say that it is not interesting. They can also say that it contradicts their intuition which means it must be wrong.

The full process is very subjective. That leads to the situation when there are groups of scientists united in coalitions that support each other and reject papers of others. I think this is what the author of the article was talking about and I can confirm that it is quite common in Economics.




.

This is a problem not with economics only, but also with other academic fields of research

Spiny Norman
24-12-2009, 04:35 AM
Relating to a argument about flagrant errors in a recent peer-reviewed article in the British Medical Journal (BMJ):


“... the BMJ is well aware that its peer review process is flawed,” says Spagat. “A recent study, whose authors include the journal’s current editor, revealed that, on average, only a third of the ‘major errors’ deliberately inserted in a BMJ article were picked up by reviewers.”

Rincewind
24-12-2009, 07:44 AM
Relating to a argument about flagrant errors in a recent peer-reviewed article in the British Medical Journal (BMJ):

This is only really surprising if you assume that the peer review peocess finds all errors in a manuscript. As already noted in this thread it doesn't. Further as a means of finding deliberately misleading "errors", peer-review works very poorly.

In the study you (sort of but not really cite, it can be read here (http://resources.bmj.com/bmj/about-bmj/about-bmj/evidence-based-publishing/What%20errors%20do%20peer%20reviewers%20detect.pdf )) the reviewers did pick up around 1/3 of the errors and so the process did improve the quality of the material that went through the process.

I do note however that the barrow-pushing people like O'Leary at Uncommon Descent (http://www.uncommondescent.com/peer-review/peer-review-life-death-and-the-british-medical-journal/) are trying to make the argument that peer-review is flawed therefore intelligent design in true. Where he says in part...


In what other line of work would such incompetence be accepted? Would you like your electrician to achieve only this level of competence? He only “gets” one third of the electrical safety hazards in your home?

However, the peer-review process is not designed to find all errors in every paper. And further his electrical analogy breaks done because there are other processes (like letters to the editor, communication between researchers and the reproduction of results by other researchers) which also find errors and are more suited to finding deliberately misleading statements. If peer-review was the only quality check than I would have to agree that one third is not many.

O'Leary finishes that article with a classic piece of wishful thinking...


Anyway, the intelligent design controversy is hardly the only area where peer review can merely maintain a convenient consensus – or tweak beards in a politically correct way.

I'm not sure if that was ment to be self-satire but I found it very amusing.

Yes peer-review does stop completely unscientific clap-trap from being published because the errors in ID articles one sees on the web or trotted out in their in-house pseudoscientific magazines are are so obvious they can hardly be missed. Their methodology is "we know the truth so lets find the fact that we can say supports that and add epicycles to explain everything that doesn't". Instead of asking "which hypothesis best explains all the observations?"

"Science" is a lot easier when you already "know" the answers. I guess that is the attraction. But since peer-review is an inconvient hurdle for the ID conspiracy theorists, and so they do try to put it down as much as possible.

Spiny Norman
26-01-2011, 08:58 AM
I subscribed to this journal some time ago, but I haven't received anything yet:

http://www.math.pacificu.edu/~emmons/JofUR/


The JofUR solicits any and all types of manuscript: poetry, prose, visual art, and research articles. You name it, we take it, and reject it. Your manuscript may be formatted however you wish. Frankly, we don't care.

After submitting your work, the decision process varies. Often the Editor-in-Chief will reject your work out-of-hand, without even reading it! However, he might read it. Probably he'll skim. At other times your manuscript may be sent to anonymous referees. Unless they are the Editor-in-Chief's wife or graduate school buddies, it is unlikely that the referees will even understand what is going on. Rejection will follow as swiftly as a bird dropping from a great height after being struck by a stone. At other times, rejection may languish like your email buried in the Editor-in-Chief's inbox. But it will come, swift or slow, as surely as death. Rejection.

antichrist
27-01-2011, 12:38 AM
The JofUR solicits any and all types of manuscript: poetry, prose, visual art, and research articles. You name it, we take it, and reject it. Your manuscript may be formatted however you wish. Frankly, we don't care.

After submitting your work, the decision process varies. Often the Editor-in-Chief will reject your work out-of-hand, without even reading it! However, he might read it. Probably he'll skim. At other times your manuscript may be sent to anonymous referees. Unless they are the Editor-in-Chief's wife or graduate school buddies, it is unlikely that the referees will even understand what is going on. Rejection will follow as swiftly as a bird dropping from a great height after being struck by a stone. At other times, rejection may languish like your email buried in the Editor-in-Chief's inbox. But it will come, swift or slow, as surely as death. Rejection.

AC
it is obviously a furfy by the editor

antichrist
17-12-2014, 01:57 AM
http://www.skepticink.com/incredulous/2014/12/12/science-denialism-skeptic-conference-redux/

Interesting because about evolutionary psychology

Rincewind
17-12-2014, 12:26 PM
It doesn't have anything to do with peer-review unless you call Edward Clint's article an example of peer-review of Rebecca Watson's talk, which in some sense it was.

Adamski
17-12-2014, 09:30 PM
I am pleased to report that the only article I have ever submitted to an academic journal was published - but after a peer review process that lasted about a year. The original article was rejected but with suggestions for changes which would make it more acceptable. You can find my article on Governor FitzRoy's debentures and their part in his sacking as Governor of New Zealand in the New Zealand Journal of History.
This article was the conclusion of a several year study originally intended to be an MA thesis but never completed due to marriage and the subsequent death of our twins- which substantially changed our priorities.

MichaelBaron
18-12-2014, 02:54 PM
Peer-review is a standard process for all of the academic publications so whether we like the process or not - it is unavoidable

Kevin Bonham
23-04-2017, 03:51 PM
https://arstechnica.com/science/2017/04/107-cancer-papers-retracted-due-to-peer-review-fraud/

107 cancer papers retracted due to peer-review fraud in which the author of the paper uses a fake email address to nominate a crony (or potentially themselves) as the peer-reviewer and there is no effective peer review.

The smoking gun? The fake reviews tended to be submitted on time!

MichaelBaron
02-05-2017, 12:46 AM
https://arstechnica.com/science/2017/04/107-cancer-papers-retracted-due-to-peer-review-fraud/

107 cancer papers retracted due to peer-review fraud in which the author of the paper uses a fake email address to nominate a crony (or potentially themselves) as the peer-reviewer and there is no effective peer review.

The smoking gun? The fake reviews tended to be submitted on time!

Not that I am surprised to read this :). Too many people take ''publish or perish'' approach to an extreme :)

Kevin Bonham
20-05-2017, 09:25 AM
A Sokal-style hoax against "gender studies" called "The conceptual penis as a social construct", complete with fake references generated by an online postmodern gibberish generator, has been pulled off against a supposed social science journal.

Fake paper here: https://web.archive.org/web/20170519175310/https://www.cogentoa.com/article/10.1080/23311886.2017.1330439.pdf

Authors' explanation here: http://www.skeptic.com/reading_room/conceptual-penis-social-contruct-sokal-style-hoax-on-gender-studies/

Nice work chaps.

Capablanca-Fan
20-05-2017, 01:21 PM
KB beat me to it!

Rincewind
21-05-2017, 01:43 PM
Authors' explanation here: http://www.skeptic.com/reading_room/conceptual-penis-social-contruct-sokal-style-hoax-on-gender-studies/

Nice work chaps.

The journal was pretty new but a reputable publishing house so a solid hit.

Capablanca-Fan
22-05-2017, 08:40 AM
Here is a defence of that hoax against critics by Helen Pluckrose (https://areomagazine.com/2017/05/21/sokal-affair-2-0-penis-envy-addressing-its-critics/), who is "critical of postmodernism and cultural constructivism which she sees as currently dominating the humanities."

Kevin Bonham
25-05-2017, 07:08 PM
http://www.perthnow.com.au/news/western-australia/the-perth-dog-thats-probably-smarter-than-you/news-story/a4de0d201ce420e0302c69532a399419

Dog appointed to the editorial boards of seven medical journals and asked to be a reviewer.

"Several journals have published on their websites a supplied photo of Dr Doll, which is actually of a bespectacled Kylie Minogue."