In an earlier essay (‘ Research Shows…’ September 7th ) I wrote of the work of John Ioannidis, in his celebrated ‘Why Most Published Research Findings are False’. The take-home messages from his work for me were two, and I quote them from the paper’s summary:
(i) ‘for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias’; and
(ii) ‘it is more likely for a research claim to be false than true’.
His study was in medical science, though the first quote is suggestive of the whole research endeavour. Well, another recent paper has added some fuel to this fire. Arturo Casadevall, of the Albert Einstein College in New York, has studied the cases where a published paper in an academic journal has been retracted later, and looked at all the cases in medically focussed research. There are millions and millions of papers in this field — something like 15 per cent of all published papers could fit under that title. Fortunately for Dr Casadevall, they all exist on a giant website.
Why is a paper retracted? The usual reason is that something was wrong — in the methodology, or the data, or the argument — and the wrong was serious enough for the conclusions of the paper to be invalid. In such a case the journal issues a notice to the effect that the paper in question has been withdrawn, with an explanation of the cause.
Casadevall pursued the explanations. He found 2047 retractions, and his first discovery was that the rate of retraction has risen sharply over the past decades. In 1976 only ten papers per million were retracted, while in 2007 the rate was 96 per million.
The main cause was fraud, and fraud plus plagiarism account for two thirds of the retractions. Casadevall is worried by all of this, not just because fraud and plagiarism are beyond the pale in academic circles, but because they cause society to lose faith in research itself. The measles epidemic that Sydney is experiencing is a sign that too many parents are not ensuring that their children are vaccinated. One reason may be a published study years ago that suggested that autism was related to childhood vaccination. That study was later retracted, and the reason was fraud, but damage was done — in the wider community there is a worry about vaccination, a fear that was not there a generation ago. There is ignorance, too, but I would guess that the fear that something bad may happen is important.
What has produced the increase in retractions? Casadevall argues that research has a disproportionate reward system, in which researchers strive to get their work published in the most cited journals, and some will do what it takes to get there. It is a highly competitive system, and rules will be bent. The same is plainly true in sport, where the use of performance-enhancing drugs is apparent, and punished when found. Why do they do it? The rewards are great if you’re not found out.
More generally, research is an industry where practitioners depend on grant money. Gaining further grants is commonly dependent on publication, and publication in the best journals. Ioannidis has argued that this is the key: ‘an obsession with winning funding has gone a long way towards weakening the reliability of medical research’.
More generally still, if the research funders have an agenda, the researchers will pursue that agenda with great care and skill. Governments are far and away the largest funders of research in every country, not just in Australia, and they have agendas. The most conspicuous example of that to me is the research that is commissioned to support the view that anthropogenic global warming is real, and is dangerous. This is the government’s view, and it is the reason we have a carbon tax.
To the best of my knowledge little or no work is being funded to explore the possibility that no matter what effect humans are having on the climate there is a strong underlying pattern of what we call ‘natural variability’, mostly because we don’t yet know exactly what causes it.
As I said in my September 5th post, I support the research endeavour, but it believe it needs some strong, independent and authoritative guardians. Papers like those of Ioannidis and Casadevall show plainly that all is not right in the research world.