Summary: Computer models have opened a new era across the many fields of science. Our confidence in their forecasts has opened a new era in scientists’ ability to influence public policy. Now come the questions. How can we determine the accuracy and reliability of these models, knowing when their forecasts deserve our confidence? What can make them more useful? {1st of 2 posts today.}
“The criterion of the scientific status of a theory is its falsifiability, or refutability, or testability.”
— Karl Popper in Conjectures and Refutations: The Growth of Scientific Knowledge(1963).
“Probably {scientists’} most deeply held values concern predictions: they should be accurate; quantitative predictions are preferable to qualitative ones; whatever the margin of permissible error, it should be consistently satisfied in a given field; and so on.”
— Thomas Kuhn, The Structure of Scientific Revolutions (1962).
About predictions
Thomas Kuhn explained that predictions are the gold standard that often decides the winner between competing scientific theories (“paradigms”). The Structure of Scientific Revolutions described failed predictions that undermined the current dominant paradigm (the Michelson–Morley experiment) and successful predictions that helped establish new paradigms (the orbit of Mercury).
With the increasing prominence of science in public policy debates, the public’s beliefs about theories also have effects. Playing to this larger audience, scientists have developed an effective tool: computer models making bold forecasts about the distant future. Many fields have been affected, such as health care, ecology, astronomy, and climate science. With their conclusions amplified by activists, long-term forecasts have become a powerful lever to change pubic opinion.
Unfortunately, models are vulnerable to confirmation bias in their construction and selection (a problem increasingly recognized, for example in the testing of drugs). Equally problematic are issues of measuring their reliability and — more fundamentally — validation (e.g., falsification).
Peer-review has proven quite inadequate to cope with these issues (which lie beyond the concerns about peer-review’s ability to cope with even standard research). A review or audit of a large model often requires over a man-years or more of work by a multidisciplinary team of experts, the kind of audit seldom done even on projects of great public concern.
Two introductions to computer modeling
(1) The climate sciences have the highest profile and most controversial use of computer models for forecasting. For a look at their use see “Questioning the robustness of the climate modeling paradigm” at Judith Curry’s (Prof Atmospheric Science, GA Inst Tech) website, discussing a paper by Alexander Bakker. His conclusion…
The paradigm that GCMs are the superior tools for climate change assessments and that multi-model ensembles are the best way to explore epistemic uncertainty has lasted for many decades and still dominates global, regional and national climate assessments. Studies based on simpler models than the state-of-the-art GCMs or studies projecting climate response outside the widely accepted range have always received less credence. In later assessments, the confirmation of old results has been perceived as an additional line of evidence, but likely the new studies have been (implicitly) tuned to match earlier results.
Shortcomings, like the huge biases and ignorance of potentially important mechanisms, have been routinely and dutifully reported, but a rosy presentation has generally prevailed. Large biases seriously challenge the internal consistency of the projected change, and consequently they challenge the plausibility of the projected climate change.
Most climate change scientists are well aware of this and a feeling of discomfort is taking hold of them. Expression of the contradictions is often not countered by arguments, but with annoyance, and experienced as non-constructive. “What else?” or “Decision makers do need concrete answers” are often heard phrases. The ’climate modelling paradigm’ is in ’crisis’. It is just a new paradigm we are waiting for.
Professor Curry has written about the problematic aspects of the current generation of global coupled atmosphere-ocean models, and ways to improve them:
- Model structural uncertainty – are GCMs the best tools?
- Broadening the portfolio of climate information
(2) For a useful introduction to the subject — and recommendations — see this article by two ecologists: “Are Some Scientists Overstating Predictions? Or How Good are Crystal Balls?“, Tom Stohlgren and Dan Binkley, EcoPress, 28 October 2014 — Excerpt:
We found a particularly enlightening paper with the enticing title, “The good, the bad, and the ugly of predictive science“. It explained a sort of common knowledge in the field that the foundation of large-scale predictable relationships was full of tradeoffs. The authors remind us that any mathematical or numerical model gains credibility by understanding the trade-offs between:
- Improving the fidelity to test data,
- Studying the robustness of predictions to uncertainty and lack-of-knowledge, and
- Establishing the “prediction looseness,” of the model. Prediction looseness here refers to the range of predictions expected from a model or family of models along the way.
… Given the … large and unquantifiable uncertainties in many long-term predictions, we think all predictions should be:
- stated as hypotheses,
- accompanied by short-term predictions with acceptance/rejection criteria,
- accompanied by simple monitoring to verify and validate projections,
- carefully communicated with model caveats and estimates of uncertainties.
Conclusions
We have few — and often no — alternatives to forecasts by computer models. They are an essential tool to guide us through an increasingly complex world. Their successful use requires more thought about ways to validate them. Too often their results appear in the news with little more than exhortations to “trust us”. That’s not enough when dealing with matters of public safety, often requiring vast expenditures.
The good news: there are immediate procedural changes scientists can take today to improve the testability of their results. Human nature being what it is, they’ll not do so without outside pressure.
Appendix: other predictions
For an example of the problem see Edwin O. Wilson’s forecast in The Diversity of Life
There is no way to measure the absolute amount of biological diversity vanishing year by year in rain forests around the world, as opposed to percentage losses, even in groups as well known as the birds. Nevertheless, to give an idea of the dimension of the hemorrhaging, let me provide the most conservative estimate that can be reasonably based on our current knowledge of the extinction process. I will consider only species being lost by reduction in forest area, Even with these cautious parameters, selected in a biased manner to draw a maximally optimistic conclusion, the number of species doomed each year is 27,000.
Here are other examples of blown or exaggerated predictions. Although not all made using computer models, they show the powerful impact of authoritative statements by scientists — magnified by activists. The common elements are that those involved remain unapologetic about their errors, there is little cost to made predictions, and few of those involved show signs of learning from their mistakes (probably because there was no cost).
- “13 Worst Predictions Made on Earth Day, 1970“, Jon Gabriel, FreedomWorks, 22 April 2013.
- “Embarrassing Predictions Haunt the Global-Warming Industry“, Alex Newman, The New American, 12 August 2014.
- Other examples of models forecasting unrealistically high rates of extinction.
For More Information
If you liked this post, like us on Facebook and follow us on Twitter. See all posts about forecasting and about computer models. Also see these articles…
- “The good, the bad, and the ugly of predictive science“, F. M. Hemez and Y. Ben-Haim, 4th International Conference on Sensitivity Analysis of Model Output (2004).
- “A tentative taxonomy for predictive models in relation to their falsifiability“, Marco Vicenconti, Philosophical Transactions of the Royal Society A, 3 October 2011.
- “Can we trust climate models?” J. C. Hargreaves and J. D. Annan, Wiley Interdisciplinary Reviews: Climate Change, July/August 2013.
- “A model world“, Jon Turney, Aeon, 16 December 2013 — “In economics, climate science and public health, computer models help us decide how to act. But can we trust them?”