Page 1 of 3 123 LastLast
Results 1 to 15 of 35
  1. #1
    CC Grandmaster
    Join Date
    Nov 2005
    Location
    Melbourne
    Posts
    5,530

    Integrity of peer-review process

    ClimateGate article in The Australian touches a very serious problem:

    Nevertheless, peer reviewing worked for many years as a more or less adequate system of quality control. In the end, the damage caused by cliquishness tended to be overcome through debate and the triumph of scientific integrity. But the situation has changed. Unfortunately, in some disciplines peer reviewing has become politicised. The way peer review is now used in public debate as a form of divine revelation - where we are told peer-reviewed science shows we must believe and do certain things - indicates how this institution risks being corrupted by advocacy researchers.
    For private coaching (IM, four times VIC champion) call or SMS 0417519733
    Computer tells you what to play. Good coach explains why.

  2. #2
    CC Grandmaster Spiny Norman's Avatar
    Join Date
    Sep 2004
    Posts
    4,437
    I am really glad this is being discussed publicly now in the mainstream media. Peer review has unfortunately become something skin to "the prophets spake thusly" and we are all expected to fall into line. Its Scientism, not Science. It has become both a political and a philosophical tool that is used by some (not many, not all, just some) to prop up faltering belief systems.
    “As you perhaps know, I haven't always been a Christian. I didn't go to religion to make me happy. I always knew a bottle of port would do that. If you want a religion to make you feel really comfortable, I certainly don't recommend Christianity.” -- C.S.Lewis

  3. #3
    Reader in Slood Dynamics Rincewind's Avatar
    Join Date
    Jan 2004
    Location
    The multiverse
    Posts
    21,567
    Quote Originally Posted by The Snail King
    I am really glad this is being discussed publicly now in the mainstream media. Peer review has unfortunately become something skin to "the prophets spake thusly" and we are all expected to fall into line. Its Scientism, not Science. It has become both a political and a philosophical tool that is used by some (not many, not all, just some) to prop up faltering belief systems.
    I think you are going just a bit too far there. There has never been any claim that peer-review was flawless and always works. Indeed there have been a number of cases where peer-review has failed in the past and will again in the future. However, it is a vital part of the operation of science and sorts out the chaff from the wheat.

    In particular if a scientific paper is written without all the facts (concealing unfavourable results) or with manipulation of the data, that is something which peer-review has a very low chance of detecting. Fortunately what will find out these results is other independent groups verify the results post-publication.

    A famous case where this happened was that of the (ex-)physicist Jan Hendrick Schön (his doctoral degree was revoked due to misconduct) who got away with making up results for a number of years. In the end, something like 20 papers had to be withdrawn by Science, Nature and Physical Review.

    The thing that concerns me about Climate Science is that I have the perception that only a (relatively) small number of people are working in the field and a reasonably large portion of the research is advocacy funded. If true, that is a recipe for politicisation of the discipline.

    PS From the link in my sigline...

    Does peer review detect fraud and misconduct?

    Peer review is not a fraud detection system. Referees are likely to detect some wrongdoing, such as copying someone else’s research or misrepresenting data, because they care about their subject. They know what research has been conducted already and the kinds of results that are likely. However, if someone deliberately sets out to falsify data, there is sometimes no way of knowing this until the paper is published and others in the scientific community scrutinise and try to repeat the work.
    So einfach wie möglich, aber nicht einfacher - Albert Einstein

  4. #4
    Monster of the deep Kevin Bonham's Avatar
    Join Date
    Jan 2004
    Posts
    36,988
    Unfortunately in the climate-emails case what is being discussed is not fabrication of data by researchers, but political interference in the refereeing process. This sort of interference can occur in various ways. Journal editors can deliberately send papers to hostile reviewers who will recommend that the papers be rejected, or biased reviewers can reject papers for bogus reasons.

    Some years ago one of my colleagues attempted to have a paper published concerning the spread of bumblebees in Tasmania (bumblebees were introduced here in 1992 and are now abundant). A seemingly harmless subject but the debate is actually contaminated by vested interest because bumblebees are excellent for tomato pollination and therefore some tomato growers would like them to be widely introduced. A major barrier to this is the counter-argument that bumblebees will go feral and interact adversely with native insects. So for a while there was a denialist line that bumblebees were adapted to European vegetation and would not forage in native forests, which as it turned out was bulldust.

    The upshot of all this is that when my colleague submitted his paper backed by records taken from all over the state by dozens of scientists (including me) the paper was rejected because one of the referees had stated that in his view the paper's results were not credible and had been fabricated. The journal got cold feet and declined to publish.

    This sort of political interference was defeated by submitting the same draft to another journal but with all the records contributors who were willing to have their names thus included listed as authors. Thus, instead of one author the paper now had dozens and no editor would take seriously that all these people would be engaged in bumblebee-data fraud.

    Typically the competition for the prestige of publishing important results first takes care of this sort of nonsense sooner or later.

  5. #5
    Reader in Slood Dynamics Rincewind's Avatar
    Join Date
    Jan 2004
    Location
    The multiverse
    Posts
    21,567
    Quote Originally Posted by Kevin Bonham
    Unfortunately in the climate-emails case what is being discussed is not fabrication of data by researchers, but political interference in the refereeing process.
    Sounds like a bit of both to me.

    Quote Originally Posted by Kevin Bonham
    Journal editors can deliberately send papers to hostile reviewers who will recommend that the papers be rejected, or biased reviewers can reject papers for bogus reasons.
    Yes all of that does happen but unless there is a world-wide conspiracy, reasonable work should be publishable somewhere.

    Regarding the bumblebee study, was it submitted elsewhere?
    So einfach wie möglich, aber nicht einfacher - Albert Einstein

  6. #6
    Monster of the deep Kevin Bonham's Avatar
    Join Date
    Jan 2004
    Posts
    36,988
    Quote Originally Posted by Rincewind
    Regarding the bumblebee study, was it submitted elsewhere?
    Yes, and published.

  7. #7
    CC Grandmaster
    Join Date
    Nov 2005
    Location
    Melbourne
    Posts
    5,530
    Quote Originally Posted by Kevin Bonham
    Unfortunately in the climate-emails case what is being discussed is not fabrication of data by researchers, but political interference in the refereeing process. This sort of interference can occur in various ways. Journal editors can deliberately send papers to hostile reviewers who will recommend that the papers be rejected, or biased reviewers can reject papers for bogus reasons.
    This is bound to happen in any area that became ideological. The further particular discipline from science the easier it's achieved.
    For private coaching (IM, four times VIC champion) call or SMS 0417519733
    Computer tells you what to play. Good coach explains why.

  8. #8
    CC International Master TheJoker's Avatar
    Join Date
    Aug 2007
    Posts
    1,646
    Quote Originally Posted by Igor_Goldenberg
    This is bound to happen in any area that became ideological. The further particular discipline from science the easier it's achieved.
    IMO Climate Change science has more in common with economics and market analysis than typical science. Being that all the findings are essentially based on regression analysis of historic data to predict future trends. There is no possibility to conduct a controlled scientific experiment to verify or test hypothesis/theory.

  9. #9
    CC Grandmaster
    Join Date
    Nov 2005
    Location
    Melbourne
    Posts
    5,530
    Quote Originally Posted by TheJoker
    IMO Climate Change science has more in common with economics and market analysis than typical science. Being that all the findings are essentially based on regression analysis of historic data to predict future trends. There is no possibility to conduct a controlled scientific experiment to verify or test hypothesis/theory.
    That problem is magnified by two factors:
    1. Unreliability of historic data pre-dating precise measurements
    2. Lack of knowledge of statistics generally and regression analysis particularly. As a result they often don't know how to validate anything but a simple model.
    For private coaching (IM, four times VIC champion) call or SMS 0417519733
    Computer tells you what to play. Good coach explains why.

  10. #10
    CC Grandmaster
    Join Date
    Nov 2005
    Location
    Melbourne
    Posts
    5,530
    Gloabl warming alarmist site published an interesting article about peer-review being necessary but not sufficient. They picked one problem - that bogus paper can be published with peer-review mechanism (no wonder they try to illustrate it with climate-sceptics articles).
    They missed the other - genuinely good high quality paper can be rejected for the same reasons - editor and/or reviewer have agendas the article does not support.
    For private coaching (IM, four times VIC champion) call or SMS 0417519733
    Computer tells you what to play. Good coach explains why.

  11. #11
    Monster of the deep Kevin Bonham's Avatar
    Join Date
    Jan 2004
    Posts
    36,988
    Peer review is basically a glorified form of argument-from-authority. It is usually the case that the peer-review process eliminates junk, but not necessarily. It's also possible for valid findings to be unfairly blocked in peer review although as Rincewind points out this form of obstruction tends not to last very long before the papers are published elsewhere, if the conclusions are basically valid.

    I have seen far too much crap in my own field get through peer review. Indeed in my own field there is no real difference in quality between a lot of the "peer-reviewed" stuff and the so-called "grey lit" (consultants' reports, theses and so on). In my own case there is a heap of stuff I have done that is just not worth the effort of writing up for peer review unless someone wants to pay me to do it, but that doesn't make it any better or worse than the stuff in the 20+ peer-reviewed papers I have published.

    I was looking at the International Code of Zoological Nomenclature recently to examine the requirements for describing a species name so that it is validly available to other taxonomists. Intriguingly, although there are requirements that a description must be issued in a form consisting of a number of copies produced simultaneously and available to the public either for free or at cost, there does not seem to be anything there that says a species name must be peer-reviewed. In theory one can self-publish; the question then is whether other scientists will subjectively decide to accept and use the names in question or not.

  12. #12
    CC Grandmaster
    Join Date
    Apr 2006
    Location
    Melbourne, Australia
    Posts
    11,094
    My academic experience makes me smile when i think about ''validity'' of peer-reviews.

    Firstly, even if review is carried out without revealing the authors' names it is usually easy to work out the authors' names based on the list of references used and reviewers knowledge of the subject matter experts.
    Secondly, Some people are ''expected'' to get a good review even if the paper is written primarily by their co-authors (in other words, their students write for them ) and their name serves as a shield to protect the paper from potential attacks.

    Thirdly, an academic review process is regarded as valid if the paper is sent to at least 2 different referees. Should only 1 of the reviews be negative, it does not stop the publisher/conference organizer from accepting the paper.

    Finally, a paper is reviewed for a publication....there is always a possibility of rejection/paper being returned to the authors with suggestions for further improvements but conference papers rarely get rejected as conference organizers want as many people as possible to attend .

    And as a bonus point, I would like to note that academics/subject matter experts usually either work together or go to another extreme due to cut-throat competition. Consequently, they hate/love each other religiously and objectivity of peer reviews can hardly be regarded as realistic to achieve.
    Interested in Chess Lessons?
    Email webbaron!@gmail.com for more Info!

  13. #13
    Reader in Slood Dynamics Rincewind's Avatar
    Join Date
    Jan 2004
    Location
    The multiverse
    Posts
    21,567
    Quote Originally Posted by MichaelBaron
    My academic experience makes me smile when i think about ''validity'' of peer-reviews.
    Out of interest, in which discipline did you have these experiences?
    So einfach wie möglich, aber nicht einfacher - Albert Einstein

  14. #14
    CC Grandmaster
    Join Date
    Apr 2006
    Location
    Melbourne, Australia
    Posts
    11,094
    Quote Originally Posted by Rincewind
    Out of interest, in which discipline did you have these experiences?
    Business, IT and communications
    Interested in Chess Lessons?
    Email webbaron!@gmail.com for more Info!

  15. #15
    CC International Master
    Join Date
    Jul 2005
    Posts
    2,104
    Quote Originally Posted by TheJoker
    IMO Climate Change science has more in common with economics and market analysis than typical science. Being that all the findings are essentially based on regression analysis of historic data to predict future trends. There is no possibility to conduct a controlled scientific experiment to verify or test hypothesis/theory.
    Very good point. The huge difference between Math and Economics (I am not even talking about its derivatives: Finance, Bussiness, etc.) is that in the majourity of Math journals the only what the referees and editors are concerned about is whether the results are correct. In Economics they also are concerned if the results are interesting, which makes everything very subjective. Imagine you are receiving your paper back with "Yeah, it looks right but the majourity of the readers will not find it interesting".

    The quatative result is that the acceptance rate for most journals in math is not less than 50%. In Economics the acceptance rate for a good journal is 10%, for a top journal it is less than 5 %.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Does God Exist?
    By Paul S in forum Religion and Science
    Replies: 5602
    Last Post: 18-07-2018, 09:19 PM
  2. Origins - Evolution, Intelligent Design and Creationism
    By Spiny Norman in forum Religion and Science
    Replies: 789
    Last Post: 01-06-2017, 06:43 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •