Ten rules to help us understand research findings

Regular readers will know that I have become disappointed with The Conversation, a publicly funded website that gives academics an opportunity to put forward their ideas and opinions about just about anything. It seems to me to be infected with what I have called elsewhere ‘the ABC culture‘, a point of view in which the planet is in trouble, women meet glass ceilings everywhere, all boat people are genuine refugees from political terror, poverty is caused by rich people, biodiversity is disappearing everywhere, and so on. And the Comments section is populated by small groups of acolytes who congratulate the author(s) of the essay and defend their point of view against all comers.

Nonetheless a summary of The Conversation appears on my screen every morning during the week, and I scroll down to see if there is anything worth reading. A couple of days ago I came across an essay on understanding researchHave you ever tried to interpret some new research to work out what the study means in the grand scheme of things? All the time, I thought, and read on. It’s a good piece, with one great howler, and I’ll summarise it for you.

1. Wait! That’s just one study If you base your views on just one study you’re making a great mistake, You’re either cherry-picking or falling for the exception fallacy. Indeed so, and I said much the same a couple of times last week.

2. Significant doesn’t mean important  Yes, ‘significant’ has a particular meaning in statistics, and readers can often drift into the view that an apparently significant finding, where p<0.001, must therefore be important. It may not be so, because in a study with a large N you are likely to get all sorts of significant relationships that have not much meaning. As a postdoc in the USA I ventured the view in a seminar that a result was ‘interesting’, and was pounced on at once. ‘What’s your theory?’ Why did you think it was interesting? What relationship did you have in mind? Why did you have it? Was it important?

3. And effect size doesn’t mean useful In medical research you might encounter a treatment that lowers the risk of something by 50 per cent. But what is the risk of your developing the condition in the first place? If it is really very small, there is no point in treating everybody so that one or two have the condition improved.

4. Are you judging the extremes by the majority? Not all trends are linear, though you might think so from the number of graphs you encounter. People with very high salt intakes have a greater risk of cardio-vascular disease. But people with very low salt intakes can have a similarly high risk too. We need to remember bell-shaped and U-shaped curves.

5. Did you maybe even want to find that effect? We are all prone to confirmation bias, and need to look hard at findings that we like.

6. Were you tricked by sciencey snake oil? The authors provide one of my favourite little videos, the fabulous Turbo Encabulator, in which a straight-faced engineer rattles off a complex stream of technical nonsense. Another example: In one study, non-experts found even bad psychological explanations of behaviour more convincing when they were associated with irrelevant neuroscience information. 

7. Qualities aren’t quantities and quantities aren’t qualities While it is par for the course to attempt a mathematical account of whatever it is you are studying, numbers may not be the way to go. Human emotions don’t lend themselves to numerical treatment, and the numbers will very likely lead one astray.

8. Models by definition are not perfect representations of reality Here comes the Whoops! A common battle-line between climate change deniers and people who actually understand evidence is the effectiveness and representativeness of climate models. Oh chaps, you were doing so well, too. This is the whole of the rest of No. 8:

But we can use much simpler models to look at this. Just take the classic model of an atom. It’s frequently represented as a nice stable nucleus in the middle of a number of neatly orbiting electrons.While this doesn’t reflect how an atom actually looks, it serves to explain fundamental aspects of the way atoms and their sub-elements work.This doesn’t mean people haven’t had misconceptions about atoms based on this simplified model. But these can be modified with further teaching, study and experience.

So how does that help us with these poor deluded ‘climate change deniers’? In my experience, it is precisely the sceptics who go to evidence, and the orthodox who keep insisting that the models explain everything — which they can’t and don’t, as No. 8 says explicitly.

9. Context matters  Individual scientists — and scientific disciplines — might be great at providing advice from just one frame. But for any complex social, political or personal issue there are often multiple disciplines and multiple points of view to take into account. Yes indeed, and ‘climate change’ provides a splendid example.

10. And just because it’s peer reviewed that doesn’t make it right  Amen. Peer review is the beginning of a study’s active public life, not the culmination. Yep.

The authors are members of the Centre for Public Awareness of Science at the ANU, and they have written a series of related articles, of which the present one is the last. They wrote one on correlation and causation, too.

I think these are excellent guidelines, and they ought to be used by anyone who is interested in the whole ‘climate change’ debate — indeed in any debate in which ‘research’ is said to show this or that. I would like to say that I use them as a matter of course, because that’s the way I was trained all those years ago. And with all due modesty, I’ll add a number 11.

11. Always go to the evidence for the claim, and try to make sense of it yourself. Don’t just accept what others say.


Join the discussion 16 Comments

  • whyisitso says:

    “They wrote one on correlation and causation, too.”

    When I was studying for my accountancy diploma five decades ago, we had to study a bit of statistics (call it Statistics 101). It was no doubt rather simplistic and I never went on to Stats 102 and upwards. We were taught emphatically that correlation didn’t denote causation.

    However in recent times that “rule” has obviously been repealed. Can someone tell me when that was – I think it was sometime in the nineties?

  • RafeChampion says:

    I was not aware that the rule that correlation does not directly denote causation has been repealed.

    On the topic of models, (8) in the list, the relevant section of Garth Paltridge’s book “The Climate Caper” is required reading, especially his account of the selection of the model that was used to plan our national climate policy. http://www.the-rathouse.com/2011/Paltridge-Climate-Caper.html

    • whyisitso says:

      I was being a bit facetious in using the word ‘repealed’, Rafe. However it does seem to me that quite a lot of the arguments advanced by warmists do amount to them maintaining that correlation is overwhelming evidence of causation.

      • David says:

        Your argument that AGW is solely reliant on a statistical
        correlation is ridiculous! There is also whole body of theory called Climate science which provides a causal mechanism explaining how an increase in CO2 can cause an increase in temperature.

        • Don Aitkin says:


          He didn’t say ‘solely reliant’. He said ‘quite a lot’. My take on it is that the initial push after 1988, and embodied in Rio in 1992, was indeed based on the coincidence of a rise in temperature and the continuing rise in CO2. Yes there was a theory, and there still is, but it was the correlation in the 1980s and 1990s that looked so powerful.

          And there was no counter theory. No one much talked about ‘natural variability’ then.

          • Rick Johnson says:

            The really weird thing now is that CO2 concentrations have continued to increase, but global temperatures have flatlined. There is no correlation, but the warmerists are claiming causation. Go figure!

          • David says:

            Yes, well the reason they do that is because it also “flat-lined” as you call it, between 1940 and 1980 before continuing to rise again.

            All figured for you Rick. 🙂

          • DaveW says:

            Thanks David for the additional proof that there is no obvious relationship between atmospheric [CO2] and estimates of global temperature. Can we stop playing political games now and devote what seems reasonable in tax dollars to determining what the climate may be up to instead of wasting vast sums on a falsified hypothesis and the jackals defending its carcass? Wouldn’t it be nice to have scientists rewarded for trying to understand climate instead of pushing a beyond parody hysteria? I don’t know about you, but I’m tired of living in the Dr Strangelove world of CAGW. I get a headache trying to decide if Professor Mann if more like Major Kong or General Jack Ripper.

          • Don Aitkin says:

            How is it figured? Wouldn’t you think that if CO2 has the proclaimed effect, and it appears to have one, then there is something else that has an even bigger effect? Even more, perhaps that something can both increase and diminish the carbon dioxide effect at different times for different reasons. There is no really persuasive account of what that something else might be, or even how many elses there might be. But on the face of it, carbon dioxide is only one effect, and not obviously a really powerful one.

          • David says:

            Define “a really powerful one.”

            Yes I agree that there may be other unidentified factors that cause temperature to cycle. By all means identify and quantify these unknown possibilities. But CO2 is increasing and going to continue to increase for the foreseeable future. The
            concern about AGW is about future temperature increases.

          • Peter Kemmis says:

            Hi David

            I imagine you read the various comments on Don’s post of 6 October the other day. I was hoping to hear from you about my comment (and some other’s I quoted) on the lack of deep ocean warming that has been measured by the ARGO floats since 2005. Here is another important observation, not that you would pick it out from the paper’s abstract, which is written in that Doublespeak that is not the sole preserve of some politicians. The second line of the abstract is a classic case:

            “Here, we infer deep-ocean warming in the context of global sea-level
            rise and Earth’s energy budget between January 2005 and December 2013.”

            But the inference is quite incorrect, as it is contradicted by the measurements. They’ve concluded that 32% of the global mean sea level rise is due to warming in the upper 2000 metres, and jumped to the apparent conclusion that the remaining 68% is caused by deep ocean warming.

            David, I find the AGW proponents are sounding more and more like “The Boy who cried ‘Wolf!’ ” So many dire predictions have we heard, so much shifting of the goal posts have we observed. Tell me, what credible argument thoroughly supported by data would you need to see, to change your mind? If you say that you rely on the IPCC and their supporting climate scientists, you are doing no more than relying on authority, rather than your own intelligence, of which you have plenty.

          • David says:


            1. “…them [warmists] maintaining that correlation is OVERWHELMING evidence of causation.”

            Anyway you want to cut it that is just not correct.

            2. Just the “1980’s” and 1990’s”, really 🙂 I think you will find they are pretty interested in the correlation from 1850 to 2015 as well. 🙂

  • Mike says:

    There are many computer models and some confusion relates to that. Design of modern transport vehicles relies on on them and enables timely cost effective solutions. Even the design results are very accurate mostly the results are still tested thoroughly. For instance jet engines are designed by computer but then a prototype is built and tested to destruction. Mostly the design is right the materials stresses and how it works is well understood. The designer of an aircraft gets into murky water though with wings. Laminar flow is not well understood and only approximations can be made but never the less computer models are still very useful to design wings. These are things though of human invention and the models produce a virtual reality of it that does rather well. When we move into creating a virtual reality model of the real world the game starts to change fundamentally.

    In the early days it was found to be most difficult to record and play sound. Special purpose integrated circuits were developed to aid in the process. The fundamental reason it was and still is difficult is that the computer world is digital not analog. The analog signal must be converted to digital to record and back again to play. The digital is a stream of discrete numbers. How many numbers you have determines whether the sound is an adequate virtual reality model of the sound. This sampling rate or resolution and it is really important if success is hoped for. There are successful computer models of much bigger and more complex systems than a stream of sound.

    In the past physical models were made of harbours so that the tide and water levels could be predicted. There are virtual reality computer models now that replace the physical ones. There are at least 30 factors involved and also the harbour must be mapped into cells. How many cells gives you the resolution. Topography of the harbour bottom inflow, sun, moon, gravitational pull of water molecules are just a few of the factors. A powerful computer produces useful results. Then there are the useless computer models.

    In the USA if you want to build by the sea the is a group called army engineers who will run a computer model which builds a virtual reality model of the section of coast required. You get a certification and there you go officialdom is happy even though it fails repeatedly! The problem is very little is known how sand moves, particularly in storm conditions. I suppose it makes money and so useful to some.

    So onto climate take a very powerful computer split the atmosphere into about 5000 cells apply many rules to each cell in turn. The interactions between cells must be considered the digital nature of the computer makes all this very difficult. In reality a change in a cell will not wait to affect other parts. So a virtual reality model of the atmosphere is created and we are told of the probability of changes up to 2100. On top of that it can not be tested! What I find remarkable is that it is believed by anyone.

  • David says:

    11. Always go to the evidence for the claim, and try to make sense of it yourself. Don’t just accept what others say.
    Agree. The thing about Point 11 is that is takes time and effort to do it properly!

  • […] seems to be funded largely by universities. I’ve complained about its bias before (here, and here, for example). Its published ‘community standards’ are […]

Leave a Reply