How can we usefully make judgments about science? Part 1

Some of these essays get republished elsewhere, most frequently on On Line Opinion, and whenever the topic is ‘climate change’ the orthodox machine-gunners assemble to attack. Their ammunition consists usually of charges that I know nothing, or that since I am not a scientist I would be incapable of understanding an article in a journal, or that I only read sceptical material, or that anything I say has been thoroughly ‘debunked’ elsewhere, or that 97 per cent of scientists agree.

I tend not to respond to such attacks, but it occurred to me that it might be worth devoting an essay or two to the business of judging science, or more accurately of judging requests for money to do with scientific research. I begin with the assertion that very few scientists, at least in my experience, are in any real sense across ‘science’ as a whole. Science has become so large, and research so specialised, that there are millions of experts, but they are expert only in a minuscule part of a small section of what were, a hundred years ago, the main scientific disciplines.

Assertion two is that no responsible Minister can make sensible judgments about science proposals, and each has to rely on the advice of others. In most developed countries there is a highly-developed system for creating that advice, and it’s much the same whatever country you are talking about, save for the USA, which is so large, and politically so special, that it really has no national system.

These advice systems are built on peer review and common-sense: you ask those in the know in that area what they think of this proposal or that one, and then you sit around and make a judgment on everything in front of you. Not all proposals are about a research topic. Some are about the need for facilities, or networks, or bringing people together, as in the Co-operative  Research Centres, an idea we borrowed from Canada. Some are about getting access to the Minister. But, in one form or another, they are all finally about spending public money. The Minister’s final decision may on occasion be made without much reference to your advice.

There are things that are impractical for Australians to do, like building a huge optical telescope in our country, because we don’t have high enough mountains. There are things that are too expensive for us to do, like building a manned space station, or setting up a rival to CERN. There are also things that we really don’t have the skilled manpower for. So Australia does what it can by sending people to the big facilities, and buying time on them, and supplying special parts for them. And we are good at it, too.

I was involved in the business of judging research proposals and research policy propositions in Australia, from about 1980 to 2000, and then in Canada from about 2000 to 2010. (You can find out more here.) In all, I must have examined about two thousand requests for money, or for a facility, or for a new policy, over those thirty years. It was a wonderful experience, and it pushed me into reading science for interest, which is what university students did until the age of the teaching laboratory, in the early 1900s. I should add that had I gone down the science path at age 15 I would probably have become a geophysicist. I find geology the most interesting science of all, but that is probably because my core interest is history, and geology is the study of the history of the planet.

How could someone like me provide useful advice? Well, everyone is a novice at the start. I first had to learn the trade, through being  an apprentice, and saw what my elders and betters did, and imitated that. I learned a few really central questions. Who will cheer if you are funded? What bad things will happen if you are not funded? Are you sure that some of this has not been done somewhere else already? Where will all the scientific staff come from? How many graduate students will be involved with this facility/machine/dig/enterprise? I learned over time how to assess the answers.

Everyone who wants money writes a proposal. It is not really hard, once you have read a dozen or so (I read and pondered on 451 in 1981, my first year, on the Australian Research Grants Committee) to see where a researcher is on top of the game, and where he or she is drawing the long bow. The best researchers write simple, modest, well-written and persuasive proposals which make questioning relatively straightforward. They still have to be questioned, and the questioners have to know what they are doing. It helps if they come from a range of disciplinary areas, as was the case with ASTEC, and indeed most of the panels I have sat on or chaired.

I served on ASTEC, the Australian Science and Technology Council, for two terms and was asked at the end would I be prepared to serve as the Chairman. I declined, in part because I had become a vice-chancellor, and had one of those 24/7 jobs. But I loved the work, and think that today’s Commonwealth Government could benefit through re-inventing ASTEC.

The core element in all this is that while I would not be able to work as a bench scientist, setting up experiments in the laboratory (my knowledge of any experimental area in any natural science was and remains pretty sketchy), I could usually read and make sense of the journal articles that came from the experimental work — if it was important to the question.

If the proposal came from particle physics I would defer to the physicists about the standing of the researcher and his or her science, but if the need was for money for a facility, a machine, or whatever, the old central questions would come to the fore quickly, and I could ask them, and gather at least a decent understanding of the issue. The rest of us on the judging panels did the same.

[Part 2 in the next post.]


Join the discussion 12 Comments

  • Mike O'Ceirin says:

    Very interesting Don it gives one an idea of the process of how research is started. One thing that is always surprised me about this is that the provider of the money that is the government has no ideas of its own. It seems that it is a process where someone decides they would like to do research in a particular subject. I had naively thought the provider of the money would have to be convinced that it was worthwhile to research this for their purposes. From what you’ve written in this blog and I think elsewhere that doesn’t seem to matter.

    • Don Aitkin says:

      There are two government positions on the general issue. One is that excellence (in science, opera, literature) is worth encouraging because in the long run it is likely to enhance the lived experience of everybody (long argument here). The second is that governments often want research done on something for good reason — for example, on the marine environment.So it sets up a ‘scheme’, provides some money, and tells the scheme to get on with it. I’ve been involved with both perspectives. They’re both sensible.

      • DaveW says:

        What isn’t sensible is when Government decides it already knows the answer or range of possible answers and provides grant schemes to scientists to prove them right.

  • David says:


    As you know I am a believer in AGW. But I will make you a standing offer. If you and I could find a data set that would enable us to disprove AGW, I would publish with you in a heart beat.

    Nature here we come. 🙂

    • margaret says:

      Most folk don’t have the backgrounds of yourself and Don – most folk watch on in either bewilderment or misjudged zealotry just trying to make their own minds up on whether AGW is real (I believe it is), while the industries created by opposing expert opinions seem to flourish disproportionately without actually helping anything.

      • Don Aitkin says:


        isn’t all debate like this? You could say much the same about boat people, what is happening in the Middle East, and even the current Budget issue. It’s taken me several years of study to have formed a coherent and consistent view on AGW. There’s a lot of work involved, and I would say the same about the other issues I raised — that is, if you want to have your own well-sustained point of view.

        • margaret says:

          Yes, I don’t doubt the study and the work and that’s why I try to understand more. But I can’t do the statistical analysis. I’m interested in the ‘binary oppositions’ that as you point out, are evident in everything. If dialectic is intended to resolve differences between opposing views rather than prove that one or the other is true, I just don’t get the point of it – apart from preventing war between the opponents. Which it doesn’t succeed in anyway. So why couldn’t one side of any given argument say okay you may be right? Then what would happen? I don’t think debating should be taught in schools. It needs to be replaced by … (not sure, I’ll have to think about that).

        • dlb says:

          I would like to think sensible governments should take a more cautious middle road approach to some of these vexed issues, rather than following the shrillest calls. With time the truth of these matters will evolve and by taking the middle road it will then be then easier to change direction. No doubt the zealots would be unhappy with this approach.

    • Don Aitkin says:

      What would we be trying to disprove? That human activities have caused all the warming from t1 to t2, most of the warming, some of the warming? I think there’s a standing offer out there in the blogosphere anyway, of $30k for anyone who can disprove AGW. It isn’t at all clear what the test would be.

      For what it’s worth, my guess is that human activity has been a contribution, but that it has been outweighed by other factors, both warming and cooling. No one has yet disentangled the human ‘forcing’ from the other.

      • Don Aitkin says:

        Here is a link to the wager:

        There seem to be a number.

        • David says:

          Yes, there would be all sorts of issues to sort through. I suppose my point is that, if I could publish a ground breaking
          article that refuted AGW, I would. I would simply write I used to be a warmist, but I have changed my mind. And here is why,…

          It would be great, on so many levels. The world could stop worrying about AGW and more importantly, I would have a good publication.

          I hope that last bit did not sound too selfish 🙂

          • Don Aitkin says:

            No, not at all.It is doubtful that AGW can be refuted unless it is expressed in every respect in terms of numbers, which observations could then show were either right or wrong. And what is ‘dangerous’, let alone ‘catastrophic’?

Leave a Reply