Summary: The events following publication of a new paper about our climate illustrates how science works in our society. It shows the warts — and its self-correcting nature. The result should please everyone, both scientists and laypeople.
Science is and has always been a social process, and hence usually confused and often nasty. Now its a mainstream subject of great importance, unlike the days when Newton lectured to an almost-empty hall. But the hype about a new article is so often unrelated to its actual importance or quality. After the media blitz comes debate, as its data and methods are scrutinized. For too many years much climate science work escaped this process, but no longer. Thanks to the Internet we can watch the process of science at work.
Of special interest is the role of outsiders, non-credentialed people who have own their own developed expertise in an area of climate science. As usual with social systems, they act as disruptive elements — revealing flawed processes that insiders agree to ignore. That our society allows (if not encourages) such people is one of our strengths.
A new paper in Nature has received massive press attention (more adulation than journalism), as providing more proof of AGW: “Warming of the Antarctic ice-sheet surface since the 1957 International Geophysical Year”, Steig, E.J., D.P. Schneider, S.D. Rutherford, M.E. Mann, J.C. Comiso, and D.T. Shindell. Nature, January 2009, pp 459-462. (Abstract for Steig 2009 here; full text here).
Contents
- Initial reaction: Praise from the pews
- Surprising results from a first look at the data
- Initial results from the involvement of outsiders
- Some scientists were cautious about the study at its publication
- My conclusions
1. Initial reaction: Praise from the pews
On the FM site Oldskeptic has strongly praised it in two comments, mirroring the mainstream media applause.
“The latest, Antarctic results have come from a newer, and yes better, methodology, incorporating the only real data source that covers the whole World, satellites.” (source)
“Note that Nature is a premier, peer reviewed scientific journal, if they could not prove their work it would not have been published.” (source)
The latter assurance is a bit odd, since it appears that the code and data are not yet fully archived. In several past major climate science articles even the reviewers did not see this vital information. No word on the code, but the lead author (Steig) has said he will archive the data soon.
Update: Not so, only a few “legitimate” folks deserve to have access to “all our data, including the intermediate steps” (source). His definition of “legitimate” is very restrictive. To Steig only a very few get access to key information on which public policy will be made – policies that might determine the fate of the world (as Al Gore has told us). And a clarification: the data inputs to the study are all public, but replication is not possible with knowing how he arrived at his results (some combination of code and intermediate work results).
Update: It gets worse. Now an editor at RealClimate says “some of these data are proprietary (NASA), but will be made available in the near future”. See comment #15 below for the lead author’s statements otherwise.
Update: It appears that the folks at RealClimate have no consistent definition of “data.” The latest spin about Steig 2009 is “{T}he raw data are public; the processed data (i.e. cloud masking) are not yet, but will be in due course. so relax”. OK, you have your orders. Relax. The authors will release this vital information when they are good and ready to do so.
Quite a soap opera. Too bad that the issues are so serious.
2. Surprising results from a first look at the data
So far investigation has been made of only two stations in the networks used (42 occupied, 65 automatic stations). Steve McIntyre (of Climate Audit) discovered that it was seriously defective, and posted about the error at 4:41 pm EST, on February 1 (Super Bowl Sunday). A later comment noted another station with defective data.
This ignited a somewhat incoherent, contradictory, defensive, and vituperative series of replies on RealClimate. See here and here. I recommend skimming the comments. Nobody likes auditors!
Corrections followed by British Antarctic Survey (BAS) and Steig, with another post defending the study by Schmidt at RealClimate (here). It is fun to watch the messy process of science in action.
3. Initial results from the involvement of outsiders
(a) In 2 posts at Climate Audit (Sunday 1 February and 2 February), Steve McIntyre identified an anomaly in the data for the “Harry” automatic weather station (AWS). On Monday BAS posted a note about the error (although originally identified by McIntyre, Gavin Schmidt notified BAS on February 1 or 2). The British Antarctic Survey (BAS) corrected the error on Monday morning — fast work, but without crediting McIntyre for catching the error. (source; for a 3rd party’s view of this episode see this by Pielke Jr., Professor of environmental studies at U Colorado).
(b) A comment by Hu McCulloch at Climate Audit uncovered another anomaly in the BAS data (the “Racer Rock” site), at 1:22am on 4 February. Later that day BAS posted a note about the error.
(c) A serious problem in climate science is failure to publicly archive data, which makes replication difficult and results in multiple and often differing “grey versions” circulating for the same data. The BAS corrections would have followed this pattern, as they initially corrected the original file. Having no version control in the public data, the data actually used in the Nature paper would vanish. McIntyre alterted BAS to this problem, and they archived the original (incorrect) data.
(d) The lead author, Steig, has posted a brief note about the impact of these data corrections, concluding that the efffect is tiny. Since he has not archived his complete code, it is not possible to replicate his analysis.
There are several other lines of inquiry about this study. Here are two, as illustrations.
(i) Accuracy of the satellite data: The trend they find is quite small (o.1 degree C per year), probably much smaller then the uncertainty of the data. No error bars are given in the article. As a general reference (which might not apply to the Steig 2009 paper), NASA’s FAQ page on the Atmospheric Infrared Sounder satellite project says:
Land surface temperature is problematical, since the emissivity of bare earth will vary greatly over the 50 km diameter spot in which our retrieval is made. Our estimated uncertainty at present is 2->3 K. {bold emphasis added}
(ii) Accuracy of the surface temperature record: There are few few stations in antarctica, and few of them have long records. Also there is evidence suggests that the automatic weather stations are often buried in snow (an excellent insulator) for long periods. See this article for pictures of visits to these stations to dig them out.
No firm conclusions can be drawn yet, but finding such data problems in a study’s first cursory review is not a good indicator of the its quality. Only time will tell.
4. Some scientists were cautious about the study at its publication
Note that initial media coverage included a few cautionary notes, ignored by true believers.
“This looks like a pretty good analysis, but I have to say I remain somewhat skeptical,” Kevin Trenberth, climate analysis chief at the National Center for Atmospheric Research, said in an e-mail. “It is hard to make data where none exist. (AP; Trenberth has been a lead author for the UN’s IPCC)
“One must be very cautious with such results because they have no real way to be validated,” says atmospheric scientist John Christy of the University of Alabama-Huntsville, who was not part of the study. “In other words, we will never know what the temperature was over the very large missing areas that this technique attempts to fill in so that it can be tested back through time.” (USA Today)
Roger Pielke Sr also had some questions, posted at his website. He is emeritus professor of the Department of Atmospheric Science at Colorado State U. See his Wikipedia entry for more information. His concluding comment:
In terms of the significance of their paper, it overstates what they have obtained from their analysis. In the abstract they write, for example, “West Antarctic warming exceeds 0.1C per decade over the past 50 years”.
However, even a cursory view of Figure 2 shows that since the late 1990s, the region has been cooling in their analysis in this region. The paper would be more balanced if they presented this result, even if they cannot explain why.
5. My conclusions
(1) Scientists eventually will sort all these questions out. True believers (on both sides), who accept only science that confirms their views, will remain unaffected no matter what the result (see here for examples). It’s called confirmation bias (see Wikipedia for more on this).
For the pro-AGW fanatics, does this result from religious fervor, or self-training in doublethink. Doublethink is the ability simultaneously to have two contradictory beliefs. Such as…
Scientists are reliable guides and sources of information. Scientists who challenge my beliefs do not understand basic science; they are incompetent. (See here for examples)
The rest of us, who will pay for the efforts proposed to reduce or stop global warming, might choose to wait until some of these questions are resolved before supporting large new public policy changes.
(2) This re-enforces My “wish list” for the climate sciences in 2009, changes that would put the climate sciences on track to meet our public policy needs. Whatever the cost (although trivial compared, for instance, to global military spending), I believe these things must be done ASAP.
- Raise the standards when applying science research to public policy questions.
- Provide greater transparency of data and methods used in climate science research.
- Provide third party review of the data, analysis, and modeling is necessary.
- Rationally apply the precautionary principle.
Afterword
Please share your comments by posting below. Per the FM site’s Comment Policy, please make them brief (250 words max), civil, and relevant to this post. Or email me at fabmaximus at hotmail dot com (note the spam-protected spelling).
For information about this site see the About page, at the top of the right-side menu bar.
For more information
To read other articles about these things, see the FM reference page on the right side menu bar. Of esp relevance to this topic:
Some of the posts on the FM site about climate change:
- A look at the science and politics of global warming, 12 June 2008
- Global warming means more earthquakes!, 19 June 2008
- An article giving strong evidence of global warming, 30 June 2008
- More forecasts of a global cooling cycle, 15 July 2008
- Two valuable perspectives on global warming, 4 August 2008
- President Kennedy speaks to us about global warming and Climate Science, 7 August 2008
- Good news about global warming!, 21 October 2008
- One of the most interesting sources of news about science and nature!, 27 October 2008
- “Aliens cause global warming”: wise words from the late Michael Crichton, 15 November 2008
It is very disillusioning when you discover that the people you trusted and believed have shown themselves unreliable. Time must pass before these things can be reconciled with oneself.
.
.
Fabius Maximus replies: I understand with the feeling, but this is just business as usual on Earth. That’s why one should always cut cards, get a second opinion before surgery, and insist that results be audited by an uninvolved third party (both in pharmaceutical research, climate science, and before investing).
FM note: As usual with climate science, much of Oldskeptic’s comments are absurd. Sad how he repeats the same misinformation (or “disinformation”). Replies are interjected in the text. If Oldskeptic follows his past practice, he will not respond, but eventually will repeat these things.
“Science is and has always been a social process”. Negative.
Hard science is and always has been a system to focus talented, fallible, intelligent, flawed, creative, silly … people within a tight discipline towards increasing knowledge. And it has always been anti-social; “sorry folks the Universe does not revolve around you despite what some badly witten books say, you are a quantum bubble of virtual particles on an insignificant speck of dirt on an insignifcant speck of space, hanging around until picking nits off each other until your sun blows up, get used to it”.
*** FM: We bow, we genuflect before the God of science! This is cracked, even funny. There is a large literature showing how science operates a social process. Thomas Kuhn and Karl Popper are two of the best know names in the field. Stephen Jay Gould’s books explain this for a general audience. What’s the alternative, to see scientists as priest-kings or gods? That they cease to have human frailities when they go into the lab?
It is the only human activity that always improves. Normal behaviour, idiologies and ‘Social sciences’ do, as all human behavour does, flip/flop between competeting idiologies (basically arguments about the colour of the underwear of the angels dancing on the heads of pins), or go into ga-ga land .. anyone remember reading about men wearing tights as the peak of fashion?.
*** FM, #1: That is one of the key points of this post. It is also a reason to be careful about the current consensus or paradigm — it changes as theory and data improve. Not to be confused with papal bulls.
Your points in detail and I am assuming you mean climate science, otherwise this post will take 1 Billion words (just dealing with the dangers of Dihydrogen Monoxide would take hundreds of thousands):
“Raise the standards when applying science research to public policy questions.”
Done, the IPCC exists to pull together all the research and present to the public and policy makers a consensus view. Plus each independent centre supplies all results to their own public and
Govt’s and then there is all the other research centres and Universities all adding to the process.
*** FM, #2: Oldskeptic appears not to understand the meaning of “raise.” Nor does he cite any evidence that the IPCC raises standards — rather than summarize existing work. This is an esp bizarre comment to make right as we see the standards actually being raised by the work of non-credentialed scientists reviewing the antarctic temperature record (see here for some details). Also see here for another specific example of the poor state of the data.
“Provide greater transparency of data and methods used in climate science research. ”
Done. Download all the data from a variety of sources (Goddard, Hadley, CSIRO, etc,e tc).
Plus download your own climate models and review the code and run them youreself.
*** FM, #3: Despite how much evidence I present to show specific assertions (like this) are incorrect, he ignores them and trots the same nonesense out the next time. This is the sad thing about Oldkeptic’s comments in this area, which makes them impossible to take them seriously. Quite sad. Look at the tremendous effort being expended to replicate Steig 2009, made far more difficult by the authors’ refusal to adequately document their methods.
“Provide third party review of the data, analysis, and modeling is necessary. ”
Done. Each centre has all its work cross checked by others (inc Unis, etc) from all over the world (US, Austrlaia, Canada, UK, Japan, etc, etc). Plus independent reviewers and any papers submitted to scientific journals are indepedently peer reviewed.
FM: Don’t just make stuff up; please give links to 3rd party review of one of the major climate models, done by a team of the appropriate experts (climate, statistics, software, etc). Such a problem is esp difficult as many of these models are large bundles of spaghetti code, such as the well-known GISS ModelE (go here for attempts to understand this model). Dan Hughes comments goes to the heart of the matter:
“Rationally apply the precautionary principle.”
Done. The classic examaple was the UK Stern report working out the scientific and economic aspects (google it and read it everyone, it really is a superb piece of work).
All a work in progress, as all science is, but constantly improving.
Re Antarctic data, the important thing is that it is not cooling overall, as earlier data and so many (including me [yes, really FM]) hoped. There are at least some hotter spots, unfortunately not where we want them (inland), but closer to the coast.
Fortunatly it is not (say) +1C, otherwise then I’d say, go to bed now and pull the blankets over your head and suck your thumb, mainly because it nice, plus it is easier on the back than bending over and KYAG. But it is a ‘warning’ …… a canary coughing a bit.
Needs more work .. hey I know, lets take all the bonuses off all those Wall Street terrorists (they make BL look like an amateur) and put it into research.
The interesting thing to me about the Antarctic Steig study is that the authors appeared to create data out of whole cloth. They produced warm temperatures from areas where no temperature sensing equipment exists. They indulged in computer fantasies, in other words. Computer fantasy is not science, it is hypothesis. Yet these curious men treat the output as if it were data! Most odd.
.
.
Fabius Maximus replies: That is a somewhat exaggerated statement of Trenberth and Christy’s comments. Steig 2009 proposes to “extend” the satellite temp record back using the sketchy surface temperature record. It’s ingenious and creative. We will see how well it stands up to analysis.
FM update: see comment #15 for more information; it appears that Id was right and I was wrong.
Steig has claimed he already archived the code and data but this is not the case. See here.
.
.
Fabius Maximus replies: That overstates the case a little.
As I understand it (please correct me if I am wrong) the sources used in Steig 2009 are publicly available, but only in external datasets that change (with poor or no version control, explained here and in the following comment). Which means that over time there is no publicly available set of the specific numbers he used, making exact and authorative replication impossible. We see that with the correction to the BAS data. If McIntyre had not requested that they archive the incorrect data, people would be using different data than the Steig 2009 paper when attempting to replicate his work.
As for the code, the last I saw was that Steig links to the statistical package (RegEm) he used. The implementation of this remains unclear (see here and here for details). As McIntyre explains:
A comment to Id’s post by Chris H. gives a good summary:
I would correct the last line to read “how science should work, using best practices.” Replication is not possible for complex work like Steig 2009 without access to some combination of code and intermediate work products (see this for an example of what’s needed to replicate Steig’s processing of the satellite data)
2 Comments on Steig. 1 on his behalf; 1 to his detriment.
1) Steig is attempting to make do with a limited set of data to explain a wide range of phenomena. This leaves him with a set of unknowns greater than his set of available parameters which in practical terms, means he can’t do his job if he leaves it at using what’s really there. Hence, he uses an old-school modeler’s trick called a synthetic anneal. He’s using some sort of interpolation techniques to fill inholes on a data grid with “scientifically feasible” synthetic data so that he has enough horsepower to set up a closed-form mathematical system. Theoretically, at least, he’s doing the right thing. This isn’t optimal science, but some days you do the best you can with what you have.
2) What Steig seems to be missing (and why I’ll now turn my guns on him) is the logical set of syllogisms justifying how he set up his anneal criterion. (I’m low on words so I’ll elide details.) An anneal criterion states, “At x-distance on a continnuum between A and B, their exist physical conditions et cetera that allow me to make a strong objective guess at to what some point C will equal for some desired parameter.” He doesn’t do this well enough to pass the BS test, therefore MacIntyre started sniffing along his continnuum. So it goes.
.
.
Fabius Maximus replies: Thanks for a great comment! I agree on all points.
These two on FM’s wish list:
Provide greater transparency of data and methods used in climate science research.
Provide third party review of the data, analysis, and modeling is necessary.
aren’t just minor, but what it means to be actual science. Results MUST be reproducable. The current failure of so many fields (not just climatology, but my field, computer science, and who knows what else) to make data, code, and models available to other researchers and general review has been a travesty.
Pingback: Science Ain’t Beanbag — Antarctic Warming Edition « World News 24/365
Comment deleted — see updated version #10 below.
FM, thanks for the balanced commentary. Us humans are plagued by all kinds of flaws, including a reluctance to see (much less admit) mistakes and predilections to classify those who disagree with us as enemies while seeing ourselves as morally superior. Thus armed, we head into battle ….
Update: More errors discovered in the Antarctica temperature record
Improving the quality of the surface temperature record, of which these are tiny first steps, is imperative to better understand climatic trends.
There are now 3 pages of correction to the British Antarctica Survey data. This flurry of corrections is largely attributed to non-credentialled scientists (not amateurs), who have promptly and courteously been acknowledged by BAS. The effort continues (e.g., see here).
(1) “Corrections to AWS (Automatic Weather Stations) data” — Mentioned above, now listing 5 stations. Also they have revised the acknowlegement to read:
(2) “The following corrections have been made using data supplied by GISS” — Corrections to 6 stations, with an acknowledgement on the “Mean Monthly Surface Air Temperature Data” main page saying:
(3) “Corrections to surface station data” — A “list of errors currently being investigated.” Now containing 5, 2 of which are by Nicholas Lewis (see his comments at Climate Audit for more detail).
Update: who gets access to information necessary to replicate climate science research?
Per Eric Steig, the lead author of the new Nature study, only a few “legitimate” folks deserve to have access to “all our data, including the intermediate steps” (source). His definition of “legitimate” is very restrictive. To Steig only a very few should get access to key information on which public policy will be made – policies that might determine the fate of the world (as Al Gore has told us).
A pure mathematician and an applied mathematician take a road trip together. As they drive along, the applied guy says, “Look, a field of flowers.” The pure guy nods in silent agreement. “Look, a field with many sheep.” says the applied guy. Again the pure guy nods. “The sheep have been shorn”, says the applied mathematician. After a long pause, the pure mathematician says,”We only know they are shorn on the side facing us.” This summarizes the debate over global warming.
For balance…
The dairy industry hires an applied mathematician to optimize milk production. He spends years studying the problem, but at last, he is ready to present his results. Before a huge audience he walks to the lectern, and puts up his first slide. “Assume a spherical cow…”, he begins.
.
.
Fabius Maximus replies: Due to my well-known humor deficiency, I don’t get the relevance of either of these to the debate over climate science.
The first is one of the basic lesson in general semantics (as in “the map is not the territory”), which IMO should be taught to schoolchildren. The second is not just the punchline of a classic joke, but also the title of an excellent book by John Harte about problem-solving.
These jokes were told to me by my doctoral adviser, an applied mathematician. At the time, perturbation method, aka asymptotic stability theory, aka chaos theory, were “The next big thing”, but older physicists and mathematicians were highly skeptical of these methods. This is because the new methods used formally divergent power series expansions to “linearize” otherwise intractable physical problems. The result was that assigning error estimates to results which were bounded only by truncating the series to leading order was near impossible.
The debate was not so much semantic, as it was philosophical. Could perturbation method be justified because it had demonstrated experimental predictive power even though as a formalism it was on shaky ground? You previously gave me a link to the “Fit an elephant” remark from Fermi to Freeman Dyson. There, I think Fermi was voicing his criticism of these emerging methods when he notes that he preferred well supported “formal” methods of setting up and solving physics problems. Failure to “Show their work” the climate guys may be attempting to avoil similar criticism of the methods they are using.
.
.
Fabius Maximus replies: Thank you for the explanation; now I got it! The article you mentioned could be about climate science (it fits so well): Freeman Dyson’s classic “A meeting with Enrico Fermi“, Nature, 22 January 2004 – “How one intuitive physicist rescued a team from fruitless research.” — Excerpt:
Update: more evidence that we need large funding increases to improve the collection of temperature data
About the data from the automatic weather stations (AWS) in Antarctica, by Guillaume Dargaud (software engineer with much experience working with instrumentation in Antarctica), from his website. (this links to his CV, in Word). Excerpt from his page about the AWS data:
Notes:
* AMRC is the Antarctic Meteorological Research Center of the Space Science and Engineering Center at the University of Wisconsin-Madison
* Concordia is the fourth and newest “permanent, all-year research station on the Antarctic Plateau”; see the Wikipedia entry for details.
Update: was the data in Steig 2009 all from public sources, as claimed?
It looks like Jeff Id was correct and I was wrong. Let’s update the evidence.
Email from Steig to Steve McIntyre on January 23 (source):
From Steig’s website at the U of Washington:
From Josefino Comiso (NASA Goddard Space Flight Center), in reply to request for data by McIntyre on January 23 (source):
Steig replying to a comment at RealClimate on February 2:
Steig replying to a comment at RealClimate on February 2:
Steig replying to a comment at RealClimate on February 3:
Steig replying to a comment at RealClimate on February 3:
Gavin Schmidt (NASA Goddard Institute for Space Studies; see his bio) replying to comment #38 by by Nicolas Nierenberg on February 9, posted at RealClimate (post titled “On Replication”):
Reply to comment #63 by Nicolas Nierenberg at RealClimate on February 9:
Reply to <a href=”http://www.realclimate.org/index.php/archives/2009/02/on-replication/langswitch_lang/in#comment-112276″ rel=”nofollow”comment #136 at RealClimate on February 11:
Gavin Schmidt replying to Comment #225 by captdallas2 on February 16 at RealClimate:
OK, you have your orders. Relax. The authors will release this vital information when they are good and ready to do so.
Congratulations to Jeff Id, Steve McIntyre, and others for some great detective work — working to find the sources the data used in Steig 2009 so that the results can be replicated. Esp noteworth is their persevance. Needless to say, this is just one more step in the process, and it is far too early to draw conclusions.
Four years later… my first visit to FM. Lookin’ good.
I have a problem with Point (4) of your wish list, namely the precautionary principle. All right, we know that the greenhouse effect is good physics, so it’s not as if the AGW people were demanding we sacrifice our first born sons to ward off rabid unicorns.
However, the action that the AGW people demand isn’t far short of sacrificing our first born sons. Billions of the poorest people around the world will have to bear the cost of the precautionary principle. The cost is out of all proportion to any possible downside.