I am not a statistician, and have never taken a course in the subject since my undergraduate days, though I have run seminars for postgraduate students about the uses of statistics in political science. I am best described as a counter, the sort of person who, while waiting for his meal in a restaurant, counts the number of diners, estimates the average cost of meals, estimates the number of employees and their average pay, and comes to a conclusion about the long-term survival of the restaurant. That’s arithmetic rather than statistics.
Over the years I’ve developed a degree of scepticism about the use of statistics in social science and elsewhere, notably climate science. I’ve written about some of that scepticism on this website, and it was what made me almost instantly suspicious about the forecasts of doom, when I began reading widely in the area around ten years ago. For example, I couldn’t believe that scientists thought you could talk seriously about changes to a temperature anomaly given to three decimal places when the accuracy of the instrument was at best one decimal place.
A couple of weeks ago, Judith Curry’s ‘Week in Review’ post sent me off to a number of other sites, one of them run by Matt Briggs, called Statistician to the Stars. I’ve been there before. Briggs (William M.) is a former meteorologist who did a PhD in statistics at Cornell, and has served as a professor of statistics. His interests are enormous, wider than mine, and he writes well, too.
What exercised him was a dispute in the Netherlands between sceptics and the orthodox. You can read it all here. The title, ‘Netherlands Temperature Controversy: Or, Yet Again, How Not To Do Time Series’ says it all, though I think the moral is wider than simply time series. I can’t do his essay justice in the space available here, because he used lots of stuff from the Dutch controversy. But I can summarise the several lessons he offers for those who want to use statistics to make their point. Here they are.
Lesson 1 Never homogenize.
Every time you move a thermometer, or make adjustments to its workings, you start a new series. The old one dies, a new one begins. If you say the mixed marriage of splicing the disjoint series does not matter, you are making a judgment. Is it true? How can you prove it? It doesn’t seem true on its face. Significance tests are circular arguments here. After the marriage, you are left with unquantifiable uncertainty.
Lesson 2 Carry all uncertainty forward.
If you make any kind of statistical judgment, which include instrument changes and relocations, you must always state the uncertainty of the resulting data. If you don’t, any analysis you conduct “downstream” will be too certain. Confidence intervals and posteriors will be too narrow, p-values too small, and so on.
Lesson 3 Look at the data.
The data are what you have (this is DA speaking). Don’t ignore outliers — they’re telling you something. What is it? Don’t homogenise (see lesson 1). Jennifer Marohasy has been criticising the Bureau of Meteorology for doing this (see, for example, here), and she is right to do so.
Lesson 4 Define your question.
Everybody is intensely interested in “trends”. What is a “trend”? That is the question, the answer of which is: many different things. It could mean (A) the temperature has gone up more often than it has gone down, (B) that it is higher at the end than at the beginning, (C) that the arithmetic mean of the latter half is higher than the mean of the first half, (D) that the series increased on average at more or less the same rate, or (E) many other things. Most statisticians, perhaps anxious to show off their skills, say (F) whether a trend parameter in a probability model exhibits “significance”.
All definitions except (F) make sense. With (A)-(E) all we have to do is look: if the data meets the definition, the trend is there; if not, not. End of story. Probability models are not needed to tell us what happened: the data alone is enough to tell us what happened (see lesson 3).
Lesson 5 Only the data are the data.
To create an anomaly is to replace the data with something that isn’t the data. It is common to take the average of each month’s temperature from 1961-1900 and subtract them from all the other months. What makes the interval 1961-1990 so special? Nothing at all. It’s ad hoc, as it always must be. What happens if you change this 30-year-block to another 30-year-block? There are all sorts of possibilities, and they can give you different answers.
Which is the correct one? None and all — what was your question again? (see Lesson 4). And that’s just the 30-year-blocks. Why not try 20 years? Or 10? Or 40? You get the idea. We are uncertain of which picture is best, so recalling Lesson 2, we should carry all uncertainty forward.
Lesson 6 The model is not the data.
The model most often used is a linear regression line plotted over the anomalies. Many, many other models are possible, the choice subject to the whim of the researcher. If you want to make a point about the data, you will find the model that does that best. But the model is not the data…
Now for the shocking conclusion. Ready?
Usually time series devotees will draw a regression line starting from some arbitrary point… and end at the last point available. This regression line is a model. It says the data should behave like the model; perhaps the model even says the data is caused by the structure of the model (somehow). If cause isn’t in it, why use the model?
But the model also logically implies that the data before the arbitrary point should have conformed to the model. Do you follow? The start point was arbitrary. The modeler thought a straight line was the thing to do, that a straight line is the best explanation of the data. That means the data that came before the start point should look like the model, too…
But they mostly don’t …
Read the original — it is great fun. And have a look at his ‘Fallacies’, too. I’ll do a post on them one day.