Oracles and Science: The Trouble with Predictions

Mortagne-reeds and elms refracted in the ball
“Mortagne-reeds and elms refracted in the ball” by Mo is licensed under CC BY 2.0

We all want to know the future, but what is the best way to predict what will happen? Assuming we don’t have a crystal ball or a time machine, we have to find patterns in the available information and use that make our best, informed guess. This is what scientists do.

There is spectrum of things we like about science. For one, science discovers fascinating things. On the more practical side though, people want science to provide reliable predictions. These predictions help people solve or avoid problems. This isn’t that different from the oracles of the ancient world. The difference, however, is that scientists need to be very systematic and open about how they came to make their predictions.

As diligent as a scientist may do these things, there are two nagging problems when it comes to the public making use of the information in scientific predictions. First, we know that there is usually variability in a system, which could lead to the actual result deviating from what was predicted. Through statistics, scientists try to account for this variability and produce quantified descriptions about our confidence in our predictions. Unfortunately, these aren’t always very useful because of our difficulty in processing numerical probabilities. The second problem is that the problem of induction could potentially pop up and surprise us. When this problem does rear its ugly head, it compromises our predictive ability, including the reliability of our descriptions about confidence.

Let’s start by discussing some of the troubles we have with understanding probability. One of the problems is that it is easy to believe something with a small probability is not likely to happen. Even with an extremely small probability, if there are sufficient opportunities for the event to occur, then it becomes likely to happen sometime. Another example of how our intuition interferes with understanding probability is our tendency to believe past experience affects the likelihood of future events. In gambling it isn’t uncommon to think that a number that hasn’t appeared for awhile becomes more likely to appear in the future. It would be equally fallible to assume that because a pair of dice summed to seven for the past ten rolls, the next roll will also sum to seven. Compounding issues like these is our tendency to misperceive probabilities by overly focusing on anecdotes. Because we tend to filter out less dramatic events, it is very easy for us to form confirmation biases. Having those biases can lead to someone even rejecting the results of more systematic data analysis because of conflicts with their perceived understanding.

It should also be noted that the resolution of a prediction and our confidence in it are inversely related. The oracles were infamous for providing vague predictions, but this increased their odds of being right! To use the classic dice as an example, I can more confidently predict that the next roll of two dice will sum to between five and nine, than I could predict the next roll will sum to exactly seven. As scientists serving society, we try to give the highest resolution predictions possible with the information we have. The marker for our limitations in that resolution is generally the acceptable uncertainty or risk.

Even if everyone grasped what the estimated probability of something meant, there is still the issue of not knowing if we know everything. As Donald Rumsfeld described it, “there are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don’t know. But there are also unknown unknowns. There are things we don’t know we don’t know.” This may sound like some kind of philosophical loop, but it’s really a practical recognition of our limitations to predict what we will find in the future. For example, say your friend wants to have lunch with you and have tomato soup. You suggest a certain restaurant because you make the prediction that it would have the best, most fresh tomato soup that day. You make this prediction because it is Thursday, and for the past two years, every time that you’ve gone to that restaurant on Thursdays the soup of the day was the best, most fresh tomato soup. But when you and your friend arrive, you find the soup of the day is chicken soup; there is no tomato soup. Was your prediction flawed? Not really. Based on the information you had, your prediction fit the pattern. Unfortunately, there were factors that you were not aware of. Maybe there was a shortage of tomatoes that week or the chef’s child was sick last night and they didn’t have time to boil down the tomatoes. This is an illustration of the ‘problem of induction.’ Essentially, induction is a great procedure for formulating ideas about how our world works, but it can’t account for situations that we haven’t encountered yet. So should we just give up on predicting the future? Probably not, but we do need to keep things in perspective.

If the public does not understand the process by which scientists make predictions, we risk scientists being viewed as oracles. For a time this could be a positive relationship, but we know in science that eventually there will be surprises. To someone who understands the scientific process, surprises are exciting opportunities to improve understanding. However, to someone who is relying on scientists’ predictions like one would the predictions from an oracle, only one mistake is enough to be discarded as a false prophet. Thus it is important for everyone to understand that as good and useful science is, there is always a chance of a prediction based on current scientific knowledge to be wrong. This should not reduce the confidence we put into science. Instead it should simply moderate our perception of it. If science didn’t lead to reliable predictions, we wouldn’t have the technology that has produced the quality of life we enjoy today.

For more on related topics, take a look at:

Bonus: Here is an example of the vague language I think should be avoided when making predictions. Maybe the audience doesn’t like hearing the possibility that a prediction can be wrong, but avoiding analysis and using overly vague language helps no one. Tips from The Enterprise System Spectator.

Accuracy vs Precision

Scientists often measure and predict things. Therefore, we need ways to describe how much we know, how close a number is to reality, and how likely we are to get the same number again. The terms accuracy and precision are generally used to describe these things, but there can be some ambiguity. This post explains the difference between the two and explores the different aspects of precision’s multiple meanings.

accuracy v precision
These targets illustrate different combinations of accuracy and precision. Accuracy describes the proximity to the desired answer (the bull’s eye). Precision describes the points’ proximity to each other. Less scatter is higher precision.

Accuracy refers to the correctness of a measurement or prediction. The results can vary a lot, but what matters is the difference between the measurements or predictions to what is considered to be the real or accepted value. Precision is often contrasted with accuracy by emphasizing the repeatability meaning of the word. This is most applicable to measurements, but can be applied to modelling too (e.g., stochastic models). When measuring something, we want some confidence that if we were to measure the same thing again, we would get a similar answer. In this situation, precision describes the degree of this similarity.

Accuracy and precision are both highly desirable characteristics for our measurements and predictions, but they are usually independent. For example, most models should produce the same results when based on the same algorithms and inputs, but this has nothing to do with the accuracy of the model’s predictions. Measurements can also be precise without being accurate. When this situation occurs, the difference between the measurements and the true value is called the bias (i.e., bias of an estimator). If we can quantify the bias, then it is possible to adjust for the bias to improve the accuracy.

Another meaning for precision creates some ambiguity when using the term. This second meaning of precision describes how much we know about a measurement. For example, rules of significant digits help us to use numbers that don’t express more than what we really know about a measurement. This definition of precision is mostly used to describe the exactness of measurements, but is sometimes used to describe the level of detail in spatial applications. For example, the popular term ‘precision agriculture’ is trying to emphasize the extra level of detail that the work can be done.

Confusion between precision and accuracy is fueled by the common mixing of the terms. Merriam-Webster defines precision as “the accuracy (as in binary or decimal places) with which a number can be represented.” Although it’s really a misuse of the term ‘accuracy’ to define a meaning of precision, they are trying to get at the concept of exactness. However, ambiguity exists in the scientific community too. The ISO has advocated defining accuracy to encompass both trueness and precision.

In order to avoid ambiguity, using alternative terms for precision may be a wise choice. When describing the ability to get similar results from multiple measurements, using the term ‘repeatability’ would be clearer in meaning. Precision, in terms of describing the quantitative or spatial detail of something, isn’t as easily replaced by a term such as ‘exactness’ because even that can conjure ideas of accuracy. As semantics evolve, maybe the meaning of precision will evolve to something more limited. In the meantime, when possible, we can use ‘detail’ and ‘resolution’ for spatial applications and ‘significant digits’ for quantitative applications.

For more, check out: