Summary: This month might be an inflection point in the three decades of struggle about the best policy to deal with climate change. Not the new IPCC special report, which has little new in it. A new paper questions the global temperature history records, the foundation of the debate.
“But facts are chiels that winna ding, and downa be disputed.
— “A Dream” by Robert Burns (1786).
An Audit of the Creation and Content of the HadCRUT4 Temperature Dataset
By John D. McLean, October 2018.
PhD in 2017 in Physics from James Cook U.
Preface
“This report is based on a thesis for my PhD, which was awarded in December 2017 by James Cook University, Townsville, Australia. …The thesis was examined by experts external to the university, revised in accordance with their comments and then accepted by the university. This process was at least equivalent to ‘peer review’ as conducted by scientific journals.”
His thesis is “An audit of uncertainties in the HadCRUT4 temperature anomaly dataset plus the investigation of three other contemporary climate issues“, submitted for Ph.D. in physics from James Cook University (2017). The HadCRUT dataset is a collaboration of the Met Office’s Hadley Centre and the Climatic Research Unit at the University of East Anglia.
Here are two excerpts from Dr. McLean’s report.
From the Introduction.
… The key temperature data used by the Intergovernmental Panel on Climate Change (IPCC) is the HadCRUT dataset, now in its fourth version and known as HadCRUT4. When I was an Expert Reviewer of the IPCC’s 2013 Climate Assessment report I raised questions as to whether the HadCRUT4 dataset and the associated HadSST3 dataset had been audited. The response both times was that it hadn’t.
Further indication that no-one has independently audited the HadCRUT4 dataset came early in my analysis, when I found that certain associated files published simultaneously with the main dataset contained obvious errors. Given the nature of the errors and the years in which some of the errors occurred, it seemed that they probably existed for at least five years. (At the time I notified the relevant people and the files have since been corrected.) It seems very strange that man-made warming has been a major international issue for more than 30 years and yet the fundamental data has never been closely examined. …
Almost all of the published papers about the HadCRUT4 dataset and its two associated datasets were written by people involved in the construction and maintenance of them, which hardly makes for unbiased analysis. …
Some issues in this study focus on individual situations, such as a single observation station, that would have negligible impact on global average values. Similar issues could exist elsewhere in the data and processing, perhaps less obviously, and the fact that issues can be identified at all suggests a variety of problems including lack of attention to detail and possible problems with fundamental procedures or processing. Above all they show that considerable uncertainty exists about the accuracy of the HadCRUT4.
The PhD candidature on which this work is based was funded on the normal “per candidate” basis by the Australian government and had no additional funding. The creation of this report itself had no funding whatsoever.
From the Executive Summary.
…As far as can be ascertained, this is the first audit of the HadCRUT4 dataset, the main temperature dataset used in climate assessment reports from the Intergovernmental Panel on Climate Change (IPCC). Governments and the United Nations Framework Convention on Climate Change (UNFCCC) rely heavily on the IPCC reports so ultimately the temperature data needs to be accurate and reliable.
This audit shows that it is neither of those things. More than 70 issues are identified, covering the entire process from the measurement of temperatures to the dataset’s creation, to data derived from it (such as averages) and to its eventual publication. The findings (shown in consolidated form Appendix 6) even include simple issues of obviously erroneous data, glossed-over sparsity of data, significant but questionable assumptions and temperature data that has been incorrectly adjusted in a way that exaggerates warming.
It finds, for example, an observation station reporting average monthly temperatures above 80°C, two instances of a station in the Caribbean reporting December average temperatures of 0°C and a Romanian station reporting a September average temperature of -45°C when the typical average in that month is 10°C. On top of that, some ships that measured sea temperatures reported their locations as more than 80km inland.
It appears that the suppliers of the land and sea temperature data failed to check for basic errors and the people who create the HadCRUT dataset didn’t find them and raise questions either.
The processing that creates the dataset does remove some errors but it uses a threshold set from two values calculated from part of the data but errors weren’t removed from that part before the two values were calculated.
Data sparsity is a real problem. The dataset starts in 1850 but for just over two years at the start of the record the only land-based data for the entire Southern Hemisphere came from a single observation station in Indonesia. At the end of five years just three stations reported data in that hemisphere. Global averages are calculated from the averages for each of the two hemispheres, so these few stations have a large influence on what’s supposedly “global”
Related to the amount of data is the percentage of the world (or hemisphere) that the data covers. According to the method of calculating coverage for the dataset, 50% global coverage wasn’t reached until 1906 and 50% of the Southern Hemisphere wasn’t reached until about 1950
In May 1861 global coverage was a mere 12% – that’s less than one-eighth. In much of the 1860s and 1870s most of the supposedly global coverage was from Europe and its trade sea routes and ports, covering only about 13% of the Earth’s surface. To calculate averages from this data and refer to them as “global averages” is stretching credulity.
Another important finding of this audit is that many temperatures have been incorrectly adjusted. {Technical explanation follows.} …
The overall conclusion (see chapter 10) is that the data is not fit for global studies. Data prior to 1950 suffers from poor coverage and very likely multiple incorrect adjustments of station data. Data since that year has better coverage but still has the problem of data adjustments and a host of other issues mentioned in the audit.
Calculating the correct temperatures would require a huge amount of detailed data, time and effort, which is beyond the scope of this audit and perhaps even impossible. The primary conclusion of the audit is however that the dataset shows exaggerated warming and that global averages are far less certain than have been claimed.
One implication of the audit is that climate models have been tuned to match incorrect data, which would render incorrect their predictions of future temperatures and estimates of the human influence of temperatures.
Another implication is that the proposal that the Paris Climate Agreement adopt 1850-1899 averages as ‘indicative’ of pre-industrial temperatures is fatally flawed. During that period global coverage is low – it averages 30% across that time – and many land-based temperatures are very likely to be excessively adjusted and therefore incorrect. …
Ultimately it is the opinion of this author that the HadCRUT4 data, and any reports or claims based on it, do not form a credible basis for government policy on climate ….
———————————————-
You can buy a copy of this report for $8 US. I recommend reading it!
The Met Office responds. McLean replies.
From The Australian: “Britain’s Met Office welcomes audit by Australian researcher about HadCRUT errors” by Graham Lloyd, 15 October 2018. As usual, climate science institutions blow the opportunity to boost their credibility with a strong response to challenges (that is what new scientists are supposed to do).
Here is McLean’s response at JoNova’s website. From high up in the peanut galley, both look like climate science at its worst. But that’s an amateur opinion.
My comments
“It is a capital mistake to theorize before you have all the evidence. It biases the judgment.”
— Sherlock Holmes in A Study in Scarlet, by Arthur Conan Doyle (1887).
, by Arthur Conan Doyle (1887).This report makes an astonishing claim: that one of the principle global temperature datasets has poor quality control. It seems negligent, given the importance of this data. Also, I look forward to experts on the climate data records critiquing his report. If McLean is wrong, MET-CRU can easily disprove his claims. Their response (or non-response) might tell us much about the institutions of climate science.
I hope Dr. McLean publishes this in a peer-reviewed journal, giving a higher level of review than for a dissertation. But that might be difficult for a green PhD who challenges the core climate science narrative. My guess is that only public pressure will make this possible (assuming, of course, that MET-CRU do not quickly disprove his claims).
Ten years ago there was considerable discussion about the poor quality control in the surface temperature record. I wrote about it, and made this my top recommendation: “More funding for climate sciences. Many key aspects (e.g., global temperature data collection and analysis) are grossly underfunded.” The cost would be pocket lint compared to our total spending on climate science, and well worth it. In 2009 Ross McKitrick gave a powerful call for action …
“I have often used the analogy of national Consumer Price Indexes to illustrate the ridiculous situation of the “Global Temperature” data. Each country has large professional staffs at their Stat agencies working on the monthly CPI using international protocols, using transparent methods, with independent academics looking over their shoulders weighing the various aggregation methodologies …and with historical archiving rules that allow backward revisions periodically if needed. It’s by no means perfect, but it’s a far cry from the f**king gong show we’re seeing here. …
“By contrast the Global Temperature numbers are coming from a bunch of disorganized academics chipping away at it periodically in their spare time. GISS numbers are handled (on Gavin’s admission) by a single half-time staffer, and the CRU says they’re stumped trying to find their original files back into the 70s and 80s, as well as the agreements under which they obtained the data and which to this day they invoke to prevent independent scrutiny.”
Assurances were given that the temperature datasets were reliable. Skeptics were mocked. But this new paper suggests that the problem was worse than most of us thought, and little or nothing has been done to improve what might be the most important data record used today. The other major global temperature datasets might be better, but I doubt it.
Nothing will change without public pressure. Push Congress to fund a full review and – if necessary, an update of these systems. Push Team Trump to implement these measures immediately.
About the author
McLean has 40 years experience as an IT professional. The Department of Physics at James Cook University awarded him a PhD in 2017. He has published four papers in peer-reviewed journals (links here). The most recent: “Late Twentieth-Century Warming and Variations in Cloud Cover” in Atmospheric and Climate Sciences, October 2014. See his website.
Trivia note: McLean has a mark of distinction – a page about him at Skeptical Science. These pages are crude propaganda, taking statements out of context and sneering at legitimate points of debate among scientists. For an extreme example, see their page about the eminent climate scientist Roger Pielke Sr.
Coverage of this story (will be updated)
“Claims of 70 problems found with key temperature dataset used by key temperature dataset used by climate models” by Graham Lloyd at The Australian, 8 October 2018 (gated). Excerpt …
“An audit of the key temperature dataset used by climate models claims to have identified more than 70 problems which the Australian author said made it “unfit for global studies”. Problems include zero degree temperatures in the Caribbean, 82 degree C temperatures in Colombia and ship based recordings taken 100km inland. …The audit is an extension of the PhD thesis by Dr John McLean awarded by James Cook University. Dr McLean has previously identified anomalies in the data set which were acknowledged by the Met Office and corrected.”
For More Information
See the new IPCC report: “Global Warming of 1.5 °C.” SR15 differs from AR15 on one major way: it assume +1.5°C over pre-industrial creates Armageddon. That’s odd, since we are already at 1°C over (much of that is natural warming). So McLean’s concerns about the global temperature dataset are important, since they could mean we are farther from the 1.5°C red level. Which is a political revision of a politically-chosen target. See “The Invention of the Two-Degree Target” in Der Spiegel.
If you liked this post, like us on Facebook and follow us on Twitter. For more information about this vital issue see the keys to understanding climate change and these posts about the wars over the historical temperature record…
- Important: climate scientists can restart the climate change debate – & win.
- Surprising news about trend of America’s temperature and precipitation.
- Alabama debunks the Times’ story about our warming world.
- About those headlines of the past century about global cooling….
- The facts about the 1970′s Global Cooling scare.
- Start of another swing of the media narrative – to global cooling?
- Global Cooling returns to the news, another instructive lesson about America.
- Did NASA and NOAA dramatically alter US climate history to exaggerate global warming? – Spoiler: no.
- The climate wars get exciting. Government conspiracy! Shattered warming records! Global cooling!
- Have the climate skeptics jumped the shark, taking the path to irrelevance?
Alarmists worked hard to keep you from reading this book.
Alarmists have worked long and hard to discredit Roger Pielke Jr.’s, because he tells us about the IPCC and peer-reviewed research. Things that violate the “narrative” about our imminent doom.
They really do not want you to read the revised second edition of The Rightful Place of Science: Disasters & Climate Change
“After nearly every hurricane, heatwave, drought, or other extreme weather event, commentators rush to link the disaster with climate change. But what does the science say?
“In this fully revised and updated edition of Disasters & Climate Change, renowned political scientist Roger Pielke Jr. takes a close look at the work of the Intergovernmental Panel on Climate Change, the underlying scientific research, and the climate data to give you the latest science on how climate change is related to extreme weather. What he finds may surprise you and raise questions about the role of science in political debates.”
Yes, its a moral panic based on what increasingly looks like dodgy data, complete with policy prescriptions which have not had any critical review, and which are both ineffective and have unintended consequences worse than the supposed problem.
The peer reviewed scientific studies may show the earth is warming. Though this study appears to show that the data is questionable.
But they don’t show that any of the proposed measures which people are using them to justify (biofuels, wind, solar, electric cars) are sensible or effective or have any material effect on the problem or do more good than unintended harm.
There’s a similar case in diet. Ancel Keys’ original studies were dodgy, but the consensus linked saturated fat, blood cholesterol and CHD. We then moved to substituting poly-unsaturated fat for saturated, with no scientific justification that it would work, and it didn’t, and with even less scientific justification and testing of it, we moved to a high carb and low fat diet. And it didn’t work either, diabetes and obesity resulted.
What the climate activists refuse to accept is that you have to test policy recommendations very carefully before concluding they will solve the problem, and will not do more harm than good. And that this is a completely separate and distinct thing from evaluating the scientific analysis of the problem.
Enthusiasm for the remedies also often leads not only to a refusal to critically evaluate their probable results but also to wilful blindness about the quality of the data, and both seem likely to have happened here.
I think developing solar/wind/electric cars etc. was worthy in its own right. Even if oil burning did literally nothing other than create regional temporary smog, there’s only so much, and we might as well develop the technology now rather than in forty or fifty years.
Of course, what I’m really hoping for is electric cars recharged from fusion plants.
Characteristic of the shift of the argument when you’re dealing with policy recommendations in the context of a moral panic.
The move is to argue that even if the policy is neither necessary to solve the problem nor effective in solving it, its desirable on other grounds so we should do it anyway. Electric cars may not lower emissions, but we should do it anyway because they improve local air quality. Very common in climate discussions.
Yes, maybe. But are we now saying that local air quality and not CO2 emissions is the important problem? And are we now saying that electric cars are the best solution to THAT? Its called moving the goalposts.
Watch out for the reply ‘but we can do both’! If you get it, ask, but if we have to choose just one, which should we choose?
Moral panics are a world view, policies and problem statement combined, all of whose disparate elements must be defended. Initially by arguing that the policies are related to the problem, but when its shown they are not, finding some other unrelated reason for keeping them.
henrik,
That’s an interesting thought: hysteria about apocalyptic global warming as a moral panic. That had not occurred to me, but it explains much of the madness – and denial of science by extremists.
Thanks for posting that! Certainly moral panics are a feature of our age. I think they have become more frequent (I haven’t seen research on this).
“It is a capital mistake to theorize before you have all the evidence. It biases the judgment.”
However, the following is not theorizing so it does not need to follow that premise:
Le HadCRUT est mort, vive le HadCRUD!
Jako,
“Le HadCRUT est mort, vive le HadCRUD!”
I like the reference, but I don’t understand your point.
If someone wondered why did I use this good old proclamation about change of reign — I couldn’t help to modify it a little bit hoping one would pause and think — the old HadCRUT isn’t really dead, it lives in the “CAGW Warning” to us all! And, especially now, when the “new and improved” data set will surely be used in the same bombastic GW propaganda, I had to change it… (I know, it was probably a little tiny bit rude)
I hope that audits like this one will be permitted to continue and not only into the data origins, but especially into that of their processing and into the conclusions and presentations and hockey sticks and all…
Jako,
We don’t know if McLean’s analysis will survive attempts to put it in the memory hole. My attempts to get climate scientists to tweet or post about it have been unsuccessful. He’s not in anybody’s club, which is a terminal disadvantage in today’s climate science.
If it gets attention, we don’t know if his analysis will withstand peer-review.
If his critique survives peer-review, we don’t know if anyone will pay attention. See above.
If his critique is acted upon, we don’t know the magnitude of the resulting change to the temp record. The brilliantly done SurfaceStation’s project resulted in only minor changes to the record.
Too early to do more than agitate for change.
I had a fun time with the IPCC report, mostly since I had friends who would link it to me and express despair. I usually responded with either “nothing new” or a GTM report about the “solar singularity” and typically got confusion.
The main reason I disbelieve any theory that all this climate science is some kind of dark horse for nebulously defined Commu-Nazi-ism is because it’s extremely incoherent. I figure it’s motivated by the same thing that compels people to repeatedly over and over try to explain Darwininan evolution to young-earth creationists. I don’t fully understand that impulse, my best guess is a kind of displaced missionary zeal.
SF,
I haven’t read the report yet. Interestingly, none of the climate scientists I know (quite a few) have read it either. The lone exception is Roger Pielke Jr, who said its major factual conclusions repeat those of SREX and AR5. He didn’t discuss the warnings.
The IPCC’s WGI reports have improved, each better than the last. AR5 was excellent. Doomsters pretty much ignored it, declaring it (without basis) “too conservative.” Now even journalists ignore the IPCC. Alarmists make better copy.
The old adage, “Liars figure and figures lie” comes to mind here. After having flown nearly 3,000 hours in WB-50 Weather bombers in the 1950s chasing typhoons in the Pacific and capturing Soviet Nuclear fallout, anything and everything even remotely connected to weather captivates and enthrals me. Add to the foregoing, flying support missions in C-124 aircraft to Ice Island Alpha near the North Pole during the International Geophysical Years.
“Weather” and its consequences have been our pilot, co-pilot and navigator many times far beyond human control. With flight logs to prove it, I’ve been touched and humbled by the Hand of God, St. Elmo’s Fire and countless other amazing and unfathomable atmospheric related realities and phenomena.
Our Typhoon missions were to penetrate the often violently turbulent wall clouds. Upon entering the calmer “eye” of the storm, a Dropsonde instrument was jettisoned/dropped by small parachute. That instrument transmitted by Morse Code critical meteorological readings from various altitudes on its way to the surface. https://en.wikipedia.org/wiki/Dropsonde Wikipedia is wrong in dating Dropsondes back only to the 1970s. We used them in the 1950s and those days the code was copied by an operator in the aircraft and not by a computer. For the most part, the transmissions were reliably clear and the copy reliably accurate until or unless the person receiving the code “mis-read” a character or two confusing the calculations of those ultimately analyzing the data.
“The biggest news about climate change” seems to be what McLean has claimed. Like garbled Morse Code that the Dropsonde operator may have misread, the news is bias and conjecture driven rather than driven by reliable data. According to McLean, the heretofore unaudited information is replete with human error and miscalculation … that is … if … McLean and HIS audit/study are to be believed.
Is the fact that we are now able to transverse the Arctic ice cap a result of a natural cycle or something man-made? Of course if man-made perhaps with time and effort something can be done do reverse the adverse effects. If natural … “Batten down the hatches matey!” In either case … “Batten down the hatches!”
To read about Dr. McLean’s work without diving (a fun word here) into “The Great White Con
Putting the Arctic sea ice record straight” found here http://greatwhitecon.info/ would be like eating potatoes without gravy.
Also lots of good stuff about the Arctic ice melt at this thread https://goo.gl/zN7TQq
Climate change? Sure … climate change every blessed day! As the ancient mariners would say, “Red in the morning, sailor’s warning. Red at night sailor’s delight!” Like the Old Farmer’s Almanac, both were fairly reliable predictors of short term climate change. Probably more reliable and predictable than the liars who figure and the figures that lie.
Just my take on it. s/The Ole’ Buzzard
A better Arctic sea ice “big picture” view is:
http://nsidc.org/arcticseaicenews/charctic-interactive-sea-ice-graph/
James, A “better” graph indeed. Thank you … wonderful … interesting looking and manipulating! s/The Ole’ Buzzard
James,
Yes, the National Snow and Ice Data Center interactive tool is excellent. Also useful is Cryosphere Today at Department of Atmospheric Sciences at the University of Illinois.
The bottom line: the long decline in arctic sea ice continues, as it has since the end of the Little Ice Age in the early 19th C (human influences became dominant after WWII). It has been flattish for 12 years, since 2006 (other than the extraordinary low in 2012).
Predictions of an imminent “arctic death spiral” have, like “the end of snow”, have been premature. These exaggerated claims have diminished the credibility of climate science.
Hi Larry,
Do you really think that a journal will publish a 135-page document that questions the basic temperature data used by the IPCC and most climate models? I figured there was no chance at all and that’s why it’s published the way it is.
If you go to the web page of details that’s part of the purchase system (https://robert-boyle-publishing.com/audit-of-the-hadcrut4-global-temperature-dataset-mclean-2018/ ) scroll down and you’ll find the data on which the audit was based.
Software will need to be created to verify many of the findings but some findings are simple. Download the zip file that contains the station data and look at the details for stations Apto Uto and Golden Rock Airport (files 800890 and 788580 – there’s no . or file extension but open in a text editor) and look closely at the data. For AU look at April, June, July 1978 and for Golden Rock check December in 1981 and 1984. Use the same method to examine any other station temperatures mentioned in the audit.
Also if you look at the global and hemispheric summary files (zipped together in the last file in the list of downloads) you’ll see the coverage in the second line for each year. For global data it’s percentage of the Earth’s surface, but for the hemispheres it’s percentage of that hemisphere. In these files the coverage is rounded to a whole number and while the same thing can be calculated to several decimal places by using the main HadCRUT4 datafile the difference is small.
Sure there’s a lot more to the audit than just these few points but at least these are a very simple start.
– JOHN MCLEAN
John,
(1) “Do you really think that a journal will publish a 135-page document that questions the basic temperature data used by the IPCC and most climate models?”
A/ A journal paper is shorter than PhD thesis.
B/ Journals have published papers critical of the AGW narrative.
C/ What I said in this post: “I hope Dr. McLean publishes this in a peer-reviewed journal, giving a higher level of review than for a dissertation. But that might be difficult for a green PhD who challenges the core climate science narrative. My guess is that only public pressure will make this possible (assuming, of course, that MET-CRU do not quickly disprove his claims).”
(2) “If you go to the web page of details”
It is not clear if that is the raw data submitted to HadCRUT, or the data HadCRUT uses (after quality control). Nick Stokes and Steven Mosher in the comments at WUWT.
(3) There is no point to debating this until there is an official response. And after that, McLean submits this to a p-r journal.
Hi Larry,
Steve Mosher’s claims were baseless. He was talking about CRUTEM4 dataset, which is of station data only, not HadCRUT4 which is stations + sea surface temperatures. The two datasets are related but different. The Climatic Research Unit (CRU) creates CRUTEM4 but works with the UK Met Office’s Hadley Centre, to create HadCRUT4.
I also checked the data in CRUTEM4 dataset and found similar spikes in the data in the grid cell that contains Apto Uto.
Mosher failed on two accounts. He assumed that
(a) what was documented for CRUTEM4 also applied to HadCRUT4
(b) that CRU documentation could be trusted to be accurate.
I’ve posted comments to WUWT, ending with a request for an apology, and I’ve posted a comment to Judith Curry’s blog.
JOHN MCLEAN
John,
You might be right. But that’s not a sufficient defense. The stakes are too high, and you are both junior and without institutional support. For example, I have not been able to enlist any of the climate scientists I know well to support you.
Write this up in a professional-quality rebuttal, as you were taught. Post it with a creative commons license – allowing reposting. I’ll post it. Anthony Watts will, imo. JoNova probably will. Perhaps we can get skeptic websites to do link to it. Perhaps we can get scientists’ websites, like Judith Curry’s Climate Etc to link to it, or even mention it.
God favors the big battalions. He doesn’t care who is right. Winning despite the odds requires playing at a higher level of skill.
Also, have you sent this to the Met and CRU, respectfully requesting them to review your findings?