Winter Storm Juno warns scientists not to burn away their credibility

Summary: We expect our leaders, and the scientists they consult, to warn us of threats. As NYC learned, again, that’s difficult to do. Should they error by warning too aggressively (false alarms), or too conservatively (fails to warn)? The consensus favors the former, ignoring the potentially massive cost of crying “wolf” too often. Someday you warn, but nobody listens. Loss of public confidence in science might be the big risk to avoid.

“To be persuasive we must be believable; to be believable we must be credible; to be credible we must be truthful. It is as simple as that.”
— Journalist Edward R. Murrow, testimony as the Director of USIA before a Congressional Committee in May 1963.

January has not yet ended and already we’ve had two waves of climate fear.  Last year we had only one, last Spring’s fear barrage about the coming super monster El Nino (that never appeared). 2015 began with reports that the “sweltering” 2014 was the the “hottest year ever” (eventually walked back to “perhaps”). This week we had the “snowpocalypse”.  It’s weather porn, collusion between publicity-hungry scientists, click-bait-seeking journalists, and activists.

But beneath the hype there are serious issues for climate scientists and our weather agencies. When and how do they issue warnings? Should they prioritize warnings — minimizing the number of times they failure to alert the public — or preserve their credibility by minimizing the number of false warnings?

The Snowpocalypse: the aftereffects might be bigger than the effects

New York City and NY State took strong precautions before the storm which so many meteorologists warned would be “historic”. In fact it largely missed NYC, hitting to the North.  Weather.com shows the records for New England, sloppily not stating the length of the record. The Boston Globe did better: NYC had the 6th largest snowfall in past 80 years. Update: these records are comparable only for roughly the past 20 years due to changes in measurement methods.

Now comes the aftereffects: TIME blamed Governor Christie and Mayor de Blasio for over-reacting, seriously inconveniencing NYC’s people — at a large economic cost.

The AP does an autopsy on the forecasts:

In a conference call with reporters Tuesday, a defensive Uccellini {Director, National Weather Service}, who wrote textbooks on winter storms, wouldn’t say his agency’s forecast was off. Instead, he blamed the way meteorologists communicated and said the weather service needs to do a better job addressing uncertainty. Uccellini said the agency would review those procedures and consult with social scientists to improve messaging. But Uccellini said he’d rather warn too much and be wrong, than not warn enough. He said the weather service’s predictions, and citywide closures that they prompted, made for a faster recovery. “This was the right forecast decision to make,” Uccellini said.

Private meteorologist Ryan Maue of Weather Bell Analytics slammed the public agency for ratcheting up forecast storm amounts before the system arrived, instead of telling people how uncertain it was. “The public should be upset that the forecast was blown for NYC and ask for answers” …

Ryan Maue (climate scientist, Weatherbell) says that the process was sound:

The authorities acted exactly as they should have considering the information that was provided to them by the National Weather Service and emergency planners responsible for putting the wheels in motion.

… The same level of media hype and perceived government (city, state) overreaction surrounding this blizzard in NYC may cause future complacency with warnings. However, the weather forecasters and authorities will not hesitate to issue the same emergency declarations because a False Alarm in the “safe direction” is hardly the worst possible outcome.

Judith Curry (Prof Atmospheric Science, GA Inst Tech) gave a typically excellent analysis of what happened and why: “Snowpocalypse – not“. Skipping to her conclusions…

 

The difference between a huge impact for NYC and what actually happened was a difference of about 25 km in the storm track, which is not a level of accuracy that you can expect from a weather forecast model.

… {Climate scientists and meteorologists should} Learn from your mistakes, understand uncertainty, and be prepared to state when you have low confidence in your prediction, or when there are two equally probable scenarios.

… Did the government decision makers respond appropriately to the forecasts they received?  I would say yes.   There is a hysteresis effect {the effect lags the cause} related to recent events (e.g. Hurricane Sandy, Buffalo snow storm).  If there are too many false alarms, you run into the ‘boy who cried wolf’ syndrome.

… Precautions don’t come without a price, but not heeding warnings could be associated with a bigger price.  With weather events, you have the opportunity to practice; even false alarms can be useful in this regard (and the expense of the precautions isn’t that overwhelming).  Communities get better with practice; Florida is now a lean mean machine when it comes to responding to hurricanes (but there haven’t been any landfalls since 2005!)  New York City seems to be in fighting form to reduce its vulnerability to future weather disasters.

The cost of failed predictions

I disagree with the conclusions of both Dr. Maue and Professor Curry (not their analysis) in one respect. But first, a small point. NYC and Florida learned from experience — fortunately mostly from close calls. Sandy could have been far worse (it hit NYC at below hurricane intensity). What if Hurricane Andrew (1992) or Katrina (2005) had directly hit Miami? This forced thought about the occurrence of past storms (e.g., the 1926 hurricane that hit Miami; the damage to NYC from the 1893 Hurricane and Hurricane Hazel in 1954). We need more of this; we’re not only unprepared for possible new weather in the future — we’re poorly prepared for the inevitable repeat of past weather.

I disagree about the cost of failed predictions. The immediate cost is, as they say, low. Logically then people apply the precautionary principle which encourages aggressive warnings, as it focuses on the cost of not acting when one’s analysis indicates a threat (“a False Alarm in the “safe direction” is hardly the worst possible outcome.”).

But using this tends to blind decision-makers to the costs of acting on false positives: exaggerated warnings or even false alarms. While not as immediately career-destroying as failures to act, they exert a real cost in lost credibility. Both mention this risk, but take it lightly. In fact it might be the largest long-term risk for warnings.

Credibility is in effect a capital sum increased by successful predictions — and reduced by withdrawals from failed predictions. This puts both decision-makers and scientists in the hot seat when considering warning the public, but there are better ways to do this than acting like riverboat gamblers going all in.

(1)  Geologists have long considered when to warn about earthquakes. They choose to make few specific predictions, and have shifted the responsibility for those to “independent” boards. The USGS has the National Earthquake Prediction Evaluation Council (NEPEC); California has its Earthquake Prediction Evaluation Council.

(2)  Don’t treat us like children; explain the probability of different outcomes and advise us about uncertainties. I believe that’s the only viable course.

Poorly done warnings are an increasingly serious problem, and not just in meteorology. Doctors risk the public getting epidemic fatigue after extreme predictions about the likely toll in America of AIDS, and more recently Swine Flu and and Ebola. As with climate, these were magnified by journalists — without push-back from leading doctors. Unless decision-makers and scientists change their behavior, some bad luck — a few more large blown forecasts — could crash the American public’s confidence in science. Which is already not too high, as seen in our long tradition of anti-intellectualism, our widespread belief in creationism and astrology, plus the large numbers who believe vaccines are harmful.

Better mechanisms for making warnings, and more nuanced warnings, will take more effort but can avoid what might be the largest loss. Burned bridges cannot easily or quickly get rebuilt.

For More Information

For more about the lessons of Juno: “What Does the Peer-Reviewed Literature Say About Trends in East Coast Winter Storms?“, Roger Pielke Jr (Prof of Environmental Studies, U CO-Boulder).

About forecasts:

1. Exaggerations and false predictions are good; truth is bad – about peak oil research
2. Checking up on past forecasts about climate change, a guide to the future
3. Looking back at claims to have predicted the Great Recession
4. Will we see the end of snow? — Written back when most scientists thought more warming means less snow (i.e., before this week).
5. Did anyone predict the 2008 crash? Will anyone predict the next crash?