In a recent post I mentioned a fine book by Aaron Wildavsky, and said I would return to it in due course. I knew Wildavsky’s name once I started postgraduate study, because with another young graduate student Wildavsky had written a little book on Australian politics in the 1920s. In those days nothing very much had been published on Australian politics so of course I read Wildavsky’s small piece on the 1926 referendum. He’d gone back to the US by the time I was ready to talk with him, and I didn’t finally meet him until I gave dinner in his honour in the mid 1980s, by which time he had been the President of the American Political Science Association, and was one of the leaders of the profession. My interests in political science by that time were not his, but I now regret that I didn’t read some of his earlier books when they were published.
Wildavsky died in 1993, and But is it TRUE? A Citizen’s Guide to Environmental Health and Safety Issues was devised and written in the late 1980s and early 1990s. It is an extraordinarily good book, and in it he and his colleagues demolish one such issue after another, including DDT, acid rain and global warming (I should make clear that he gave the ozone layer issue his solitary tick, while my own judgment is suspended for lack of evidence). Given that his book is now twenty-five years old, what is staggering to me is that his findings are no less robust now than they were in the early 1990s.
What I want to write about here is the notion of a citizen scientist. Wildavsky argues that the term ‘citizen scientist’ does not mean simply that someone is able to carry out scientific work independently of a university or some other research laboratory. He has in mind the scientifically interested person who, though not someone who goes to the lab bench every day, is nonetheless able to assess scientific work in terms of its utility in guiding public policy.
I agree entirely, and in my own way I have been engaged in that endeavour for a long time. So have many others who have found themselves in similar advisory positions. How should we best do it? Wildavsky argues that the citizen should learn to recognise patterns of representation so as to avoid being controlled by them. Each citizen is his or her own private eye. I like that play on words. All of us would-be risk detectives need experience in being misled and still, in the end, catching the real villain or, as the case may be the false accuser.
How should we do it? Wildavsky says that there is no substitute for reading scientific journals and learning, and I would agree. If you finally don’t understand the point, ask someone who does. These days, with the Internet, finding out about the nature of a scientific issue is probably speedier than it was twenty-five years ago. He goes on:
To be a risk detective the citizen has to be alert to the typical errors and defects in experimental studies. Scientists and government experts are apt to make mistakes in experimental design, measurement and inference… a number of methodological flaws … appear repeatedly in the scientific justification for regulatory practice. And Wildavsky derives rules to guide the citizen risk-detective in finding them. There are twenty rules, so that for several there will only be an outline.
Rule 1: Use Appropriate controls
You need control groups for some issues and appropriate baselines for others. In the ozone layer case, we really have little idea of whether or not the hole in the latter had been there before, and/or how long before. In that case and in the global warming one, so many of the effects seem to lie within the range of natural variation. How then can we substantiate claims and policy inferences?
In the case of control groups, it is important to control for age, for sex, for exposure to whatever it is, for pregnancy in some cases, and so on. What are the pre-existing conditions? It was conventional to assume that ‘pure rain’ had a pH of 5.6, and that led regulatory agencies to assume that rain with a notably smaller pH must be a sign of human activity. But then later studies showed a wide range of of pure rain pH, from 4.4 to 5.8. What then?
Rule 2: Establish the baseline
Increases and decreases do not come from nowhere. They are selected partly on the basis of existing data and convenience and partly to make whatever point the user has in mind. By knowing where the base begins, we learn interesting things: How long, for instance, have measurements been made?
Rule 3: Vary the baseline to determine whether the conclusion is robust
Choosing the Cretaceous era as a baseline, the earth today appears starved of carbon; choosing the beginning of the Industrial Revolution of the nineteenth century, the earth appears to be carbon rich or at least richer. If the conclusion is the same no matter what the baseline, it is on firmer ground.
Rule 4: Remember that parts are not necessarily wholes
Trees are not forests, and individual forests are not regional ones. Sydney is not Australia, and Australia is not the world.
Rule 5: Count what counts
If one is measuring the effects of ozone, remember that UV radiation also causes skin cancer, not only ozone loss.
Rule 6: Follow trends
It is frustrating to have to wait, but often we have to. It can take time to determine a trend.
Rule 7: Establish the normal range for the phenomenon
Rule 8: Use the same types of measurement consistently
Rule 9: Prefer measurements to estimates
And I would add, prefer them to the output of model simulations.
Rule 10: Be aware of recall bias in assessing exposure
Rule 11: Consider the duration of exposure
Rule 12: Evaluate separate effects to determine if there is really something to worry about
Rule 13: Be aware of the extrapolation of effects
It’s easy to establish the results of high doses of something, but we should not assume that bad outcomes coming from a high dose must mean that any dose is harmful.
Rule 14: Seek the mechanism and Rule 15: Establish conditions of applicability and Rule 16: Do not accept residual explanations
All these hang together. The notion that carbon dioxide must be the villain, because current science cannot come up with anything else, is a laughable position to maintain — but it has been argued this way.
Rule 17: Don’t draw final conclusions from one study
One for the ABC and the mainstream media generally.
Rule 18: Be skeptical
Learn to puncture inflated claims.
Rule 19: Keep score
Rule 20: Seek diversity, not uniformity of opinion
Dissent is really important, because it is from disagreement that new knowledge comes. (see my masthead above).
These rules are intended for the citizen scientist, the risk-detective. But surely they should apply to governments and regulatory agencies, not to mention scientific academies. All the above comes from Chapter 14 of this most important, and immensely readable book. Yes, you can buy copies of it online (mine came from Better World Books)
End-note: As I said earlier, I am moving to publishing one essay a week. The next will be in a week’s time. I will still be monitoring the website on a daily basis.