Peer review of scientific work – an inadequate basis for big public action

Summary:  After a decade of stone-walling and evasion by major scientific journals about climate science articles, I find it astonishing that so many folks regard this as a sound basis on which to construct public policy.  If the Food and Drug Administration approved drugs in this way, people would be dieing like flies from bad side-effects. 

Failure to replicate is a glitch in the academic process of science.  The truth will come out, eventually.  All that is lost is time and money.  However, since these are scare resources, both textbooks and institutional policy statements often put great emphasis on the importance of replication of research results.  Note this excerpt from “Psychology: An Introduction”, Russell A. Dewey, PhD (not esp authoritative, but nicely expressed).

Public policy requires higher standards, as lives — even nations — are at risk.  This post gives tangible evidence of the need for higher standards.  Note the contrast between those following best practices, and those slower to adopt them.  The good news is that this shows the process of science working, albeit slowly and imperfectly (like all social processes).

This is the third post discussing the level of research required for public policy action on climate change.  The first two are here and here.

The climate science debate has tested science journals as have few issues in their history.  Without public access to the data, replication is not possible.  If the reviewers themselves do not see the data, their “peer review” has little significance.   Here we look at two examples, one good — and one bad.  

More important, we see the self-correcting nature of science, as the “bad” journal evolves to copy best practices elsewhere.

Contents

1.  The good:  Philosophical Transactions of the Royal Society: Biological Sciences (aka Phil Trans B)
2. The bad:  International Journal of Climatology
3. Epilogue:  incremental but real progress
4. Another note on peer-review, from an experienced scientist

1.  The good:  Philosophical Transactions of the Royal Society: Biological Sciences (aka Phil Trans B)

Who are these guys?  From Wikipedia: 

Phil Trans, is a scientific journal published by the RoyalSociety. Begun in 1665, it is the oldest scientific journalprinted in the English-speaking world and the second oldest in the world, after the French Journal dessçavans. It is still published, making it the world’s longest running scientific journal. The use of the word “philosophical” in the title derives from the phrase “natural philosophy”, which was the equivalent of what we would now generically call “science”.

In 1887 the journal expanded to two separate publications, one serving the Physical Sciences: Philosophical Transactions of the Royal Society A: Physical, Mathematical and Engineering Sciences and the other focusing on the life sciences: Philosophical Transactions of the Royal Society B: Biological Sciences.

Excerpt from “Phil. Trans. B“, Steve McIntyre, Climate Audit, 30 December 2008:

Last summer, I reported that I had requested that Briffa archive data for Briffa et a l2008, pursuant to policies of Phil Trans B. I received a cordial note at the time from the editor that they took their policies seriously and would follow up on it. They responded cordially when I followed up on several occasions and said that they were working on it (and, given that they were dealing with core Team members, this is no small undertaking) and hoped to have the data by the end of the year.

A couple of days ago, I was notified by Phil Trans B that the first installment (Scandinavian data) was online here and that the authors had undertaken to have the balance online “in the New Year”.

All in all, a very professional response from Phil Trans B, placing the surly response from International Journal of Climatology in rather stark contrast and perhaps highlighting rather neatly the questionable data standards that are deemed acceptable by the Royal Meteorological Society and some climate scientists.

Many climate scientists have responded well to requests for more information on their data and methods, often eager to have their work subjected to critical scrutiny.  But not all…

2.  The bad:  International Journal of Climatology

This is a small but telling about the limitations of academic peer review as a basis for public policy, showing how not all journals take their peer review responsibilities equally seriously. Note the epilog, with indications of improvement at the IJC.

(a)  The article under discussion:  “Consistency of modelled and observed temperature trends in the tropical troposphere”, Ben Santer et al, International Journal of Climatology, Volume 28, Issue 13, 15 November 2008 — Abstract:

A recent report of the U.S. Climate Change Science Program (CCSP) identified a “potentially serious inconsistency” between modelled and observed trends in tropical lapse rates (Karl et al., 2006). Early versions of satellite and radiosonde datasets suggested that the tropical surface had warmed by more than the troposphere, while climate models consistently showed tropospheric amplification of surface warming in response to human-caused increases in well-mixed greenhouse gases. We revisit such comparisons here using new observational estimates of surface and tropospheric temperature changes. We find that there is no longer a serious and ubiquitous discrepancy between modelled and observed trends in tropical lapse rates.

This emerging reconciliation of models and observations has two primary explanations. First, because of changes in the treatment of buoy and satellite information, new surface temperature datasets yield slightly reduced tropicalwarmingrelative to earlier versions. Second, recently-developed satellite and radiosonde datasets now show larger warming of the tropical lower troposphere.

In the case of a new satellite dataset from Remote Sensing Systems (RSS), enhanced warming is due to an improved procedure of adjusting for intersatellite biases. When the RSS-derived tropospheric temperature trend is compared with four different observed estimates of surface temperature change, the surface warming is invariably amplified in the tropical troposphere, consistent with model results. Even if we use data from a second satellite dataset with smaller tropospheric warming than in RSS, observed tropical lapse rates are not significantly different from those in all model simulations.

Our results contradict a recent claim that all simulated temperature trends in the tropical troposphere and in tropicallapserates are inconsistent with observations. This claim was based on use of older radiosonde and satellite datasets, and on two methodological errors: the application of an inappropriate statistical “consistency test”, and the neglect of observational and model trend uncertainties introduced by interannual climate variability.

(b)  As part of his larger project of auditing climate research papers, Steve McIntryre wrote a measured analysis:  “Santer et al 2008“, 16 October 2008.  He then sent the following request (see here for full details of this correspondence):

Dear Dr Santer, Could you please provide me either with the monthly model data (49 series) used for statistical analysis in Santer et al 2008 or a link to a URL. I understand that your version has been collated from PCMDI ; my interest is in a file of the data as you used it (I presume that the monthly data used for statistics is about 1-2 MB). Thank you for your attention, Steve McIntyre

(c)  On 16 October 2008 he received this reply:

Dear Mr. McIntyre,

I gather that your intent is to “audit” the findings of our recently-published paper in the International Journal of Climatology (IJoC). You are of course free to do so. I note that both the gridded model and observational datasets used in our IJoC paper are freely available to researchers. You should have no problem in accessing exactly the same model and observational datasets that we employed. You will need to do a little work in order to calculate synthetic Microwave Sounding Unit (MSU) temperatures from climate model atmospheric temperature information. This should not pose any difficulties for you. Algorithms for calculating synthetic MSU temperatures have been published by ourselves and others in the peer-reviewed literature. You will also need to calculate spatially-averaged temperature changes from the gridded model and observational data. Again, that should not be too taxing.

In summary, you have access to all the raw information that you require in order to determine whether the conclusions reached in our IJoC paperare sound or unsound. I see no reason why I should do your work for you, and provide you with derived quantities (zonal means, synthetic MSU temperatures, etc.) which you can easily compute yourself.

I am copying this email to all co-authors of the 2008 Santer et al. IJoC paper, as well as to Professor Glenn McGregor at IJoC.

I gather that you have appointed yourself as an independent arbiter of the appropriate use of statistical tools in climate research. Rather that “auditing” our paper, you should be directing your attention to the 2007 IJoC paper published by David Douglass et al., which contains an egregious statistical error.

Please do not communicate with me in the future.

Ben Santer

(d)  McIntyre comments on this here:

My request to Santer was for the data as used by him. It’s easy to register at PCMDI. But you won’t find the data as used by Santer. I’m handy at extracting data from archives and don’t mind working at it, and the public record is pretty clear on that -so that’s not the issue.

… But y’know, I’m not going to argue about this. There are policies governing this. Let’s see what happens. I’ve sent FOI requests to the Department of Energy and NOAA; I’ve sent a request to the journal under their policies; I’ve sent a request to DOE under their mission statement. It seems to me that it would have been simpler for Santer to send me the data, but, as I say, if he wants to be a small center of attention in this corner of the blogosphere, I’ll oblige him.

I’ve had some success with this with other obstructionist climate scientists. We’ve been getting information from Jones and Briffa inch by inch under the British FOI legislation. Science magazine hated the bad press about Esper and eventually made him provide most of his data (though Thompson is too big for them to deal with.)

If the climate science community chooses to cheer for obstructionism, as Phil Jones cheered Santeron, this is not a strategy that I would recommend to them. IF they are confident in their results, then they should provide data instantly. Take this sort of issue off the table.

(e)  McIntyre also submitted a request under the Freedom of Information Act.  Four of the co-authors work for NOAA; Santer works for Lawrence Livermore Laboratory — a “private” organization funded largely by the US government.  McIntyre comments here on the need for this struggle to bring important climate science data and methods into the light:

As a start, there is a very clear policy statement requiring federal recipients to archive data after a limited period of exclusive use. If NSF enforced these policies on climate scientists that I deal with, that would go a long way to resolving some of my issues. Or if journals enforced their stated policies. To a considerable degree, NSF has become a cheerleader and has abdicated its compliance responsibilities. This happens from time to time in business situations and it seldom is to the benefit of shareholders and the public.

If you notice, I spend far more time and attention trying to get journals and agencies to administer their stated policies. I actually take more exception to mealymouthed answers from bureaucrats and editors than to obstruction by climate scientists.

(f)  The results of the FOI request were zip, as the coauthors denied they had any correspondence about data used in this article.  See here for details. 

(g)  Lastly, McIntyte went to the journal.  This has worked before, as most journals require that the basis of articles be made available for public review.  This round in the battle is described at “Data Archiving not required by the International Journal of Climatology“, Steve McIntyre, posted at Climate Audit, 28 December 2008.  Summary:

Dear Dr McGregor { Editor of the International Journal of Climatology}:

I am writing to you in your capacity as editor of International Journal of Climatology.  Recently Santer et al published a statistical analysis of monthly time series that they had collated from 47 climate models. I recently requested a copy of this data from Dr Santer and received the attached discourteous refusal.

I was unable to locate any information of data policies of your journal and would appreciate a copy of such policies.

The form of my request was well within the scope of the data policies of most academic journals and I presume that this is also the case in respect to the policies of your journal. If this is the case, I would also appreciate it if you required the authors to provide the requested collation in the form used for their statistical analysis. While the authors argue that the monthly series could be collated from PCMDI data, my interest lies with the statistical properties of the time series, rather than with the collation of the data.

{from the next letter}  Thank you for this follow up. For your information, although the Santer SI reported that the inclusion of more recent data does not affect their “H1” results, they either omitted to carry out or neglected to report the results of including more recent data on their “H2” results. I found that inclusion of data up to 2006, 2007 or to the most recent data reverses the conclusions reported in their Table III (using their own methods. I successfully emulated their Table III results for data up to end 1999). It is my intention to submit a comment to your journal reporting these calculations.

{from the next letter}  For your reference, here are policies at Nature and Phil Trans Roy Soc, both of which require provision of data. In many econometrics journals, authors are required to archive data AND working code at time of submission.

Given the use of climate articles for policy, it is vital that journals have adequate data archiving policies and that they are enforced. In my opinion, you should ensure compliance has been completed (at least in escrow) as a condition of review – this is what econometrics journals now do – as this saves rear guard actions with reluctant authors.

I would also like to comment on their H1 data but require the data already refused by Santer in order to carry out the analysis, and, if the data remains unavailable, will, of course, note this refusal in my planned submission.

(h)  His reply was as follows:

Dear Dr McIntyre;

In response to your question about data policy my position as Chief Editor is that the above paper has been subject to strict peer review, supporting information has been provided by the authors in good faith which is accessible online (attached FYI) and the original data from which temperature trends were calculated are freely available. It is not the policy of the International Journal of Climatology to require that data sets used in analyses be made available as a condition of publication. Rather if individuals are interested in the data on which papers are based then they are encouraged to communicate directly with the authors.

With this email I consider this matter closed. 

Regards, Glenn McGregor

(i)  McIntyre’s closing analysis on this affair:

My original request to McGregor was for a copy of the data policies of the journal. I guess that his answer is that there is no policy. In the present situation, I notified McGregor that Santer had already refused to provide the requested data. Now McGregor says that I am “encouraged to communicate directly with the authors”.

McGregor says that the paper was “subject to strict peer review” – as opposed, I suppose, to casual peer review. Yet the “strict peer review” was incapable of noticing that Santer et al had failed to carry out their analysis on the most recent data that they mention and failed to inquire as to the effect of such analysis.

This sleaziness is very tiresome.

The fate of the world is at stake, or so we are told — but personal pique and pettiness still limit public access to key information about the threat.  Does this suggest that many of the scientists involved do not take their own warnings seriously?

Note:  the International Journal of Climatology is a publication of the Royal Meteorological Society.

3.  Epilogue:  incremental but real progress

(a)  “A response from Prof Hardaker“, Posted at Bishop Hill blog, 5 January 2009 — Bishop Hill is not a bishop. He’s not actually called Hill either. He is an Englishman who lives in rural Scotland.  Excerpt:

(b)  With commendable speed, Professor Hardaker, the CEO of the Royal Meteorological Society has responded to my email asking for a statement on the Society’s position on one of its journals standing in the way of an attempt to replicate a study published there.

Thanks for your note. I’ve had a couple of emails relating to this discussion and the position currently is as Prof McGregor mentioned. As I have mentioned to others who have emailed in, I’m very happy to consider the requirement for a clear policy statement and as such I have put this on the agenda for the next meeting of the Society’s Scientific Publishing Committee, at which all the Editors of the Society’s journals are members. 

(c)  In response to a follow-up email, Prof Hardaker has provided some more details:

Many thanks for the pointer to the blog and I would be very happy to feed your thoughts into the discussion. I’d also be very happy to share with you the decision of the Committee, and of course any new policy would be publicised through our respective Journal pages. I would wholly expect that if the Committee decided to establish a policy it would do so for all the journals in the Society’s portfolio, but that’s just my assumption. The Scientific Publishing Committee next meets in early May. I recognise this is a little way away but it will give us the time to take a considered look at the comments we have received and best practice from other respected journals in time for when we send the committee paperwork out in April. I will also make sure the Chair of our Committee sees all the correspondence we have had associated with this.

Note that Professor Hardaker responds promptly, without question the credentials or job title of “Bishop Hill.”  These things mean little in science.  The question is valid, and he acts appropriately.

If this produces meaningful change, it will not be the first such policy change sparked by non-scientist AGW-skeptics.  But there is a long way to go before the peer-review process works adequately. 

(d)  Steven McIntyre notes (bold emphasis added):

It seems odd that they {the Royal Met Society} have no policy, but they don’t. So let’s hope that they develop one. For what it’s worth, Climatic Change instituted a data policy as a result of my acting as a reviewer.

In early 2004, in my capacity as a reviewer, I asked for supporting data and code. Schneider said that, in 28 years of the journal, no reviewer had ever made such a request.Needless to say, that didn’t impress me as a reason not to make the request. He said that it would require a decision of their editorial board, so I asked him to obtain such a decision. They then agreed on data but not code.

I then asked for supporting data under the new policy, which the authors (MBH) refused to provide and the manuscript disappeared from sight (in the mean-time, they made a check-kiting citation to it in Mann and Jones 2005, so that accomplished what they wanted.) It’s such a stupid game.

However, the exercise was not entirely pointless. I asked for data from Thompson and this resulted in the scraps of information on Dunde, Dasuopu and Guliya now available digitally.

(e)  Ross McKitrick adds this comment:

This is the situation everywhere. It comes as a surprise to people outside academia that reviewers don’t check the data and calculations. It comes as a surprise to people inside academia that anyone thought they did. Replication work does not begin until after a study has been published, and even then it is rare. When authors refuse to release their data and code it makes replication work difficult or impossible, meaning nobody checks the results ever.

4.  Another note on peer-review, from an experienced scientist

Also of interest about this:  “Q&A with Roger Pielke, Jr. on the climate change story“, CE Journal, 8 January 2008 — Pielke  is a Professor in the University of Colorado’s Environmental Studies Program and a Fellow of the Cooperative Institute for Research in Environmental Sciences. 

Note his comments on the peer review process:

Q: Have we put too much faith in the peer review system? And should we seek sources outside the usual scientific circles?

A: Peer review is simply a cursory check on the plausibility of a study. It is not a rigorous replication and it is certainly not a stamp of correctness of results. Many studies get far more rigorous peer review on blogs after publication than in journals. I use our own

blog for the purpose of getting good review before publication for some of my work now, because the review on blogs is often far better and more rigorous than from journals. This is not an indictment of peer review or journals, just an open-eyed recognition of the realities.

It is hard to say who is outside and who is inside scientific circles anymore. McIntyre now publishes regularly in the peer reviewed literature. [Pielke is speaking of Steve McIntyre, whom I would describe as a climate change gadfly; he publishes a blog called “Climate Audit“] Gavin Schmidt blogs and participates in political debates. [Schmidt is a NASA earth scientist who conducts climate research.] Lucia Liljegren works at Argonne National Lab as an expert in fluid dynamics and blogs quite well on climate predictions for fun. She is preparing a paper for publication based on her work, but she has never done climate work before. I am a political scientist who publishes in the Journal of Climate and Nature Geoscience and blogs. Who is to say who is ‘outside’ and who is ‘inside’? Is participation in IPCC the union card? How about having a PhD? Publishing in the literature? Testifying before Congress?

… This debate featuring Wigley and Holdren about who is inside and who is outside is enlightening along these lines: “Climate Experts Debating the Role of Experts in Policy“, Prometheus (a science policy blog), 8 January 2008. Tom Wigley is a climate scientist at the National Center for Atmospheric Research, and John Holdren, who is an environmental policy expert at Harvard , was tapped to be President-elect Barack Obama’s science advisor.

Afterword

Please share your comments by posting below.  Per the FM site’s Comment Policy, please make them brief (250 words max), civil, and relevant to this post.  Or email me at fabmaximus at hotmail dot com (note the spam-protected spelling).

For information about this site see the About page, at the top of the right-side menu bar.

For more information

To read other articles about these things, see the FM reference page on the right side menu bar.  Of esp relevance to this topic:

• About Science & Nature – my articles
• About Science & nature – studies & reports

Articles zbout the battle for release of climate science data and methods

1. “Data Archiving, Disclosure and Due Diligence”, Climate Audit.
2. “Fortress CRU #2: Confidential Agent Ammann“, Steve McIntyre, Climate Audit, 20 June 2008.
3. “Fortress CRU“, Steve McIntyre, Climate Audit, 20 June 2008.
4. “Fortress Met Office“, Steve McIntyre, Climate Audit, 20 June 2008.
5. “Fortress Met Office continued“, Steve McIntyre, Climate Audit, 23 June 2008.
6. “E-Mail, “Personal” Records and Privacy“, Steve McIntyre, Climate Audit, 2 July 2008.
7. “Climate Audit and NOAA FOI Policy“, Steve McIntyre, Climate Audit, 3 July 2008.
8. “NOAA Response to March 2007 FOI Request“, Steve McIntyre, Climate Audit, 8 July 2008.
9. “CSIRO adopts Phil Jones’ Stonewall Tactic“, Steve McIntyre, Climate Audit, 15 July 2008.
10. “Openness & Government“, Shane Deichman, at MountainRunner, 26 July 2008
11. “Is Briffa Finally Cornered?“, Steve McIntyre, Climate Audit, 30 July 2008
12. “Emulating Mannian CPS“, Steve McIntyre, Climate Audit, 2 December 2008 – The struggle continues to get “hockey stick” Mann’s computer to code to work.  Only then can Mann’s work be replicated.  Odd that it appears in peer-reviewed journals; one wonders what “reviewed” means when the code does not run.
13.  “Mann et al 2008 – Another Error Notice“, Steve McIntyre, Climate Audit, 5 December 2008 — Another correction by Mann, who seems unable to credit his critics.
14. Peer review of scientific work – an inadequate basis for big public action – Documentation of McIntyre’s stuggles to get key data released about 2 climate science articles.