Is the way we fund and carry out science part of the problem?

A regular correspondent sent this to me, asking did it sound familiar. And it does. You’re allowed a guess or two as to what field the writer is  talking about. It isn’t ‘climate change’, but it could be, because some of the same fundamental weaknesses in the funding and carrying out of science  are present there too. The author is American and he is plainly talking about the USA — yet how much of what he points it is not also evident here? A lot of it, I think.

The responsibility for this unfortunate state of affairs rests squarely on the leaders of XXXX research. Rather than training graduate students in the scientific method, and allowing their research to serve the needs of society, the field’s leaders choose to train their mentees to serve only their own professional needs—namely, to obtain grant funding and publish their research. I have experienced these practices myself as I transitioned from student to graduate research assistant to research fellow, and colleagues continue to emphasize that this is how it must be, lest they fail to get funding and ‘feed’ their graduate students and families. But by not training mentees in the basics of science and skepticism, the XXXX field has fostered the use of measures that are so profoundly dissonant with scientific principles that they will never yield a definitive conclusion. As such, we now have multiple generations of XXXX researchers who dominate federal XXXX  research and the peer review of that work, but lack the critical thinking skills necessary to critique or conduct sound scientific research.

The apparent self-interest that is driving research in this field is not limited to raising students to merely follow the herd. The subjective data yielded by poorly formulated XXXX studies are also the perfect vehicle to perpetuate a never-ending cycle of ambiguous findings leading to ever-more federal funding… an estimated $2.2 billion on XXXX research in the 2012 fiscal year, a significant proportion of which was spent on research that used … pseudoscientific methods … The fact that XXXX researchers have known for decades that these techniques are invalid implies that the field has been perpetrating fraud against the US taxpayers for more than 40 years—far greater than any fraud perpetrated in the private sector (e.g., the Enron and Madoff scandals).

When anti-science rhetoric occurs at a Kansas school-board fight over creationism, we can nod our educated heads in silent amusement, but if multiple generations of XXXX researchers have been trained to ignore contrary evidence, to continue writing and receiving grants, and to keep publishing specious results, the scientific community as a whole has a major credibility issue. Perhaps more importantly, to waste finite … research resources on pseudo-quantitative methods and then attempt to base public … policy on these anecdotal “data” is not only inane, it is willfully fraudulent.  

The solution to this dilemma is quite simple: funding agencies must stop funding flawed XXXX research, and the editors of … journals need to stop publishing the results. Given the immense amount of money invested in this field each year, this goal is much easier to state than to accomplish.

OK. The field is nutrition, and the lament comes from a paper that you can read here. The lead author, Edward Archer, and his colleagues looked at all the data from the National Health and Nutrition Examination Survey (NHANES) from 1971 to 2010, covering 29,000 men and 35,000 women. Archer zeroes in on the basic weakness: each respondent’s food intake was estimated from a 24-hour retrospective recall of his or her actual diet over that day. Converting the reported diet into energy values led to ‘caloric energy intake’ figures that Archer says are just ‘physiologically implausible’. In other words, respondents systematically under-reported what they had eaten.

His conclusion: the ability to estimate population trends in caloric intake and generate empirically supported public policy relevant to diet-health relationships from U.S. nutritional surveillance is extremely limited. In his view researchers are, or should be knowledgeable about the problem of self-reporting, and no less aware that the other constituents are basal energy expenditure and physical activity: it’s not only what you take in, but what is needed to keep you going plus how much physical activity you engage in.

The failure of nutrition studies to deal with such obvious weaknesses means, as he sees it, that after about 40 years and many millions of dollars of US nutritional surveillance data were fatally flawed. In most research domains, such a finding might be monumental; yet in nutrition epidemiology—the study of the impact of diet on health, here(in)after referred to simply as “nutrition”—these results are commonplace. In fact, there is a large body of evidence demonstrating that the systematic misreporting of energy and macronutrient intake renders the results and conclusions of the vast majority of federally funded nutrition studies invalid.

I can see obvious parallels with climate science: the failure of researchers to look critically at their own work and that of their colleagues, the stern defence of what they have published, the ignoring of really basic problems in the methodology, the production of results that please the paymasters, and so on.

There is a temptation to argue that it is only climate science where these glaring weaknesses exist, and to provide good reasons why this might be true. But this article, and other work by John Ioannidis that I have mentioned in the past, suggest that there may well be systemic problems in the way contemporary science is funded and carried out.

Join the discussion 8 Comments

  • Patrick says:

    Excellent post as usual Don. I think you have ‘hit the nail on the head’ with this one.
    Much more rigorous standards are needed to implement more rigorously the scientific method. Critical thinking is fundamental in the design, implementation and peer review of all scientific endeavours. My personal opinion is that logic, the scientific method and critical thinking should be part of the basic school curriculum so that the wider community is better equipped to assess published research.

  • David says:

    Don,

    I fail to see the significance of this post. Climate scientists do not use self-reported
    data.

    For a man who loves to tyre-kick other people’s scientific capabilities, you are a bit
    of a scientific lightweight. As far as I can see you have mainly published in history and political science journals. The AGW hypothesis was first proposed by Svante Arrhenius in 1896 and the theory has never been disproven. Instead, the theory has gained wide spread support. Science can never prove anything it can only present evidence consistent with a hypothesis. However it can falsify.
    So time to put up. Write a post with some supporting scientific analysis
    that falsifies AGW, instead of more of your vague unsupported claims about poor
    scientific method.

    Walk the walk!

    .

    • Don Aitkin says:

      David,

      No one has to publish scientific papers in order to be able to read them! Over the past 35 years I have read thousands of them, and where I can’t understand I ask for help, if I think the paper is important. I spent thirty years in the science funding, policy and regulation domain, and in two countries.

      Your comment seems to aim at me, rather than at what I write, which is in the tradition of scepticism and critique. If you can find what I write faulty, then by all means point it out.

  • Peter Lang says:

    David,

    You said: “Science can never prove anything it can only present evidence consistent with a hypothesis. However it can falsify.”

    The problem is that policy and ideology driven ‘science’, which is what CAGW is, has not been putting serious effort into trying to falsify the AGW hypothesis. It has been putting its effort into finding effort to support, it no falsify it. Huge amounts of public funding have been steered towards supporting the hypothesis, not to trying to falsify it. It has been skeptics from outside academia and the orthodoxy who have done the real work in falsifying key parts of the evidence. Examples are Steve McIntyre with the ‘hockey stick’ and Nic Lewis with climate sensitivity.

    More important in my opinion is that science is not directly relevant to policy analysis. It needs to go through economics, engineering, law, diplomacy, etc to get to relevant policy advice. This is what scientists have not recognised. They’ve tried to drive policy directly. So, more and more they have resorted to advocacy and adopted tactics of exaggeration, scaremongering, doomsaying to try to get attention of policy makers.

    But temperature change in itself is not a measure of costs an benefits or of danger. It is not relevant until the damages are defined. And more and more we are finding that the cost of the policies proposed so far, exceed by an enormous margin, the projected benefits.

    Luckily, there are people from outside the orthodoxy who can bring an enormous wealth of skills and experience to assist with rational analysis. This post may be of interest:

    “Confidence levels inside and outside an argument” http://judithcurry.com/2013/10/06/confidence-levels-inside-and-outside-an-argument/

    • David says:

      Peter,

      I agree with your second paragraph, that policy needs to be guided by the social sciences that you list.

      However, I don’t agree with your first paragraph. Lots of money is invested in climate science generally. There are obviously huge incentives for the fossil fuel industry to falsify AGW. Any scientist with a half-decent research proposal, which might falsify AGW, would obtain funding is a heartbeat.

      For example, Prof Richard Muller was a skeptic who was commission by US Coal industry to test the AGW. His concern was that data was being “cleaned” before it was analyzed. So his dataset consisted of 14 million raw observations drawn from 44,555 sites dating back to 1753. That was a massive undertaking to collect such a large dataset. As it turns Muller concluded that the Earth has indeed warmed and humans are the cause.

      • Peter Lang says:

        David,

        However, I don’t agree with your first paragraph. Lots of money is invested in climate science generally. There are obviously huge incentives for the fossil fuel industry to falsify AGW. Any scientist with a half-decent research proposal, which might falsify AGW, would obtain funding is a heartbeat.

        Around $100 billion has been invested in climate science and policies. Virtually all of it has been funded by government’s. The fossil fuel industry has spent very little in comparison. The government funding is spent to largely support the orthodox view not to try to falsify the hypothesis. professors do not encourage their students to disagree with them.

        Muller is poor example to choose. His history of statement reveal he was always a warmist.. But that is not a useful argument to pursue anyway. And virtually no one is arguing that the Earth has warmed since the Little Ice Age, so making that point is a strawman tactic.

        Any scientist with a half-decent research proposal, which might falsify AGW, would obtain funding is a heartbeat.

        As I am sure you recognise, it is not that easy just making a research proposal because of the enormous amount of money being spent to support the orthodoxy and the enormous disincentives to argue against it. I suspect you realise that the the CAGW hypothesis is not really a scientific hypothesis because there are no statements of what is required to falsify it. So, it is far more akin to religious belief than to a scientific hypothesis.

        Just to make clear, my scepticism and deep concern is about the claims of “Catastrophic” consequences of AGW and the costs of the policies that are being advocated. In this I responded to youjr comment here http://donaitkin.com/the-science-may-be-uncertain-but-shouldnt-we-take-precautions/#comment-1115375729

        That explains my main concern and I expect it is what is the real underlying cause for most people’s scepticism. So that is the issue that really needs to be addressed. I am really surprised at your reply as I said in my response to your comment.

        • David says:

          “So, it [AGW} is far more akin to religious belief than to a scientific hypothesis.

          Rubbish!

          “…because there no statements of what is required to falsify it.”
          Try this,

          The Null hypothesis is that an increase atmospheric CO2 has not effect on global temperature.

        • David says:

          Muller is poor example to choose. His history of statement reveal he was always a warmist..
          Show me where ??

Leave a Reply