Oracles and Science: The Trouble with Predictions

We all want to know the future, but what is the best way to predict what will happen? Assuming we don’t have a crystal ball or a time machine, we have to find patterns in the available information and use that make our best, informed guess. This is what scientists do.

There is spectrum of things we like about science. For one, science discovers fascinating things. On the more practical side though, people want science to provide reliable predictions. These predictions help people solve or avoid problems. This isn’t that different from the oracles of the ancient world. The difference, however, is that scientists need to be very systematic and open about how they came to make their predictions.

As diligent as a scientist may do these things, there are two nagging problems when it comes to the public making use of the information in scientific predictions. First, we know that there is usually variability in a system, which could lead to the actual result deviating from what was predicted. Through statistics, scientists try to account for this variability and produce quantified descriptions about our confidence in our predictions. Unfortunately, these aren’t always very useful because of our difficulty in processing numerical probabilities. The second problem is that the problem of induction could potentially pop up and surprise us. When this problem does rear its ugly head, it compromises our predictive ability, including the reliability of our descriptions about confidence.

Let’s start by discussing some of the troubles we have with understanding probability. One of the problems is that it is easy to believe something with a small probability is not likely to happen. Even with an extremely small probability, if there are sufficient opportunities for the event to occur, then it becomes likely to happen sometime. Another example of how our intuition interferes with understanding probability is our tendency to believe past experience affects the likelihood of future events. In gambling it isn’t uncommon to think that a number that hasn’t appeared for awhile becomes more likely to appear in the future. It would be equally fallible to assume that because a pair of dice summed to seven for the past ten rolls, the next roll will also sum to seven. Compounding issues like these is our tendency to misperceive probabilities by overly focusing on anecdotes. Because we tend to filter out less dramatic events, it is very easy for us to form confirmation biases. Having those biases can lead to someone even rejecting the results of more systematic data analysis because of conflicts with their perceived understanding.

It should also be noted that the resolution of a prediction and our confidence in it are inversely related. The oracles were infamous for providing vague predictions, but this increased their odds of being right! To use the classic dice as an example, I can more confidently predict that the next roll of two dice will sum to between five and nine, than I could predict the next roll will sum to exactly seven. As scientists serving society, we try to give the highest resolution predictions possible with the information we have. The marker for our limitations in that resolution is generally the acceptable uncertainty or risk.

Even if everyone grasped what the estimated probability of something meant, there is still the issue of not knowing if we know everything. As Donald Rumsfeld described it, “there are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don’t know. But there are also unknown unknowns. There are things we don’t know we don’t know.” This may sound like some kind of philosophical loop, but it’s really a practical recognition of our limitations to predict what we will find in the future. For example, say your friend wants to have lunch with you and have tomato soup. You suggest a certain restaurant because you make the prediction that it would have the best, most fresh tomato soup that day. You make this prediction because it is Thursday, and for the past two years, every time that you’ve gone to that restaurant on Thursdays the soup of the day was the best, most fresh tomato soup. But when you and your friend arrive, you find the soup of the day is chicken soup; there is no tomato soup. Was your prediction flawed? Not really. Based on the information you had, your prediction fit the pattern. Unfortunately, there were factors that you were not aware of. Maybe there was a shortage of tomatoes that week or the chef’s child was sick last night and they didn’t have time to boil down the tomatoes. This is an illustration of the ‘problem of induction.’ Essentially, induction is a great procedure for formulating ideas about how our world works, but it can’t account for situations that we haven’t encountered yet. So should we just give up on predicting the future? Probably not, but we do need to keep things in perspective.

If the public does not understand the process by which scientists make predictions, we risk scientists being viewed as oracles. For a time this could be a positive relationship, but we know in science that eventually there will be surprises. To someone who understands the scientific process, surprises are exciting opportunities to improve understanding. However, to someone who is relying on scientists’ predictions like one would the predictions from an oracle, only one mistake is enough to be discarded as a false prophet. Thus it is important for everyone to understand that as good and useful science is, there is always a chance of a prediction based on current scientific knowledge to be wrong. This should not reduce the confidence we put into science. Instead it should simply moderate our perception of it. If science didn’t lead to reliable predictions, we wouldn’t have the technology that has produced the quality of life we enjoy today.

For more on related topics, take a look at:

Bonus: Here is an example of the vague language I think should be avoided when making predictions. Maybe the audience doesn’t like hearing the possibility that a prediction can be wrong, but avoiding analysis and using overly vague language helps no one. Tips from The Enterprise System Spectator.

The Cycle of Science

In an earlier post I contrasted induction and deduction while suggesting that induction is the currently favored term used in science. However, I also suggested that the two philosophies can be used in concert with one another. Indeed, as much as one can argue about the virtues of one philosophy or the other, science actually advances on a cycle between the two!

Let’s review. To put things simply, induction emphasizes the individual, whereas deduction begins with generalizations. Science largely began with a focus on deduction, trying to find universal laws and hoping to explain as much about the world as possible. In the 19th century, generalizations reached a pinnacle of overuse, particularly for explaining human characteristics. While deduction became disfavored because of those missteps, induction rose to favor on the heels of more quantitative techniques for describing the individual objects of study.

Criticisms and attempts to further distinguish the two range from the practical limitations of the respective philosophies to the innateness of ideas. The debate over the origin of ideas about universals is as at least as old as Plato and Aristotle. For the most part, science has sided with Aristotle on the importance of experience (empiricism). In a way this explains science’s gravitation towards the principles of induction, but in the process, the problem of induction has been cast aside.

The problem of induction is the impossibility to observe every individual case. Here’s a classical quote on the issue:

When they propose to establish the universal from the particulars by means of induction, they will affect this by a review of either all or some of the particulars. But if they review some, the induction will be insecure, since some of the particulars omitted in the induction may contravene the universal; while if they are to review all, they will be toiling at the impossible, since the particulars are infinite and indefinite.

-Sextus Empiricus (160-210 CE) from Outlines of Pyrrhonism

In other words, there is always uncertainty about what hasn’t been directly observed and it is impossible to observe everything. To help fill the gaps between observations, we make generalizations that are based on what has been observed so far. In modern science these predictive guides take the form of models. There are different types of models, but one of the most popular is the empirical model. Defining a model as empirical emphasizes the fact that it is built from, or calibrated on, observed data. The purpose of the model is to then help the researcher understand system dynamics and make predictions. But the application of the model (generalization) to unknown situations is deduction.

Although the terms regularly used today are different, in practice modern science uses induction and deduction in a cycle. Data is collected, which is often used to build models. Where weaknesses in these models are found, more data is collected and different model designs are tested in order to make a better model. Thus the cycle of science is a process of continually refining knowledge. Deduction enables predictions and helps to identify exceptions to the generalized models. Induction accumulates more data to test and improve those generalized models.

For more on the philosophical controversy of induction, deduction, and all the entities in between, check out the series of posts on The Lycaeum blog.

Also, hopefully it is easy to see how the scientific method fits nicely into this philosophical model. For example, tomatosphere.org presents the scientific method as a cyclical flow chart that parallels the diagram for the cycle of science presented here.

Why Geography?

The scientific discipline of geography has taken on many important topics over the course of its history. From locating natural resources, to understanding the intricacies of human cultures in diverse locations, to predicting climate change and its impacts on society, geography has helped us better understand our world.

Students in grade schools are taught basic geographic facts such as the location of countries, because knowing where things are, is the first step to figuring out why those things are there. Geography began in a similar way; geographers documented where things were and developed better ways to measure and communicate that information. Eventually though, geographers expanded onto questions of ‘why’ are things where they are. The description on the title page of Emanuel Bowen’s 1747 book (image to the right) may reflect a beginning of geography’s transition from cataloging spatial information to trying to make sense of it (understanding the system). Today, geography continues to document where things are (hence, a strong relationship with GPS), but spends most of its time studying the interactions between spatial phenomena. Because addressing modern problems require system approaches, understanding the time and space components of the processes involved is more important than ever.

Unfortunately, geography education has not progressed with the advancement of geography as a science. For many people the study of geography ended with location facts and they have never been exposed to the fruitful endeavor of scientific geography. The opportunities to increase scientific knowledge from geographic research are so immense that the discipline of geography has struggled to define a focused mission.

In addition to limited exposure to spatial science, people’s perception of geography is also confused by the diversity of issues that geographers work on. For example, what do topics in geomorphology have in common with topics in geopolitics (both topics regularly studied by geographers)? I think the answer is that the spatial component of these topics is crucial to understanding why phenomena occurred, are occurring, or will occur. Can these topics be studied by separate disciplines, focusing on the issues by subject (e.g. Geology, Political Science)? Of course, but what geographers bring to the table is a unique perspective on the complications of studying spatial phenomenon. And these complications are only beginning to be understood.

The complications of geographic research are regularly underestimated. For example, the basic concept of the modifiable areal unit problem was identified in the 1930s, but too often research is conducted oblivious to the dependency of the results on analysis scale. This concept is the primary concept behind gerrymandering, yet its potential to bias research (especially in the natural sciences) is rarely acknowledged.

As further evidence that the details of geographic concepts are often not fully appreciated, I point to the fact that fundamental terms for describing spatial concepts are ill-defined. The loose use of geographic terms leads to confusion in the scientific literature about what is actually being studied. One of the clearest examples of this is use of the term “scale.” Confusion about this term has been exasperated by the transition from paper to digital maps. Because of the constraints of drawing a map on paper, extent and map unit size were essentially bound together. Under those circumstances, map scale described both spatial concepts at once. However, on digital maps, resolution, map unit size (~analysis scale), representative fraction (cartographic scale), and extent are independent of one another. Digital methods provide a great deal more freedom to study spatial phenomena, but we must be careful to define our methods (i.e. by ‘scale’, is one describing resolution, extent, or analysis scale).

Geography needs to continue to work on the ‘applied’ topics that it has been working on, but in the process, geography should use the information gathered to further illuminate and define the characteristics of spatial phenomena (e.g. establishing standards for describing the different aspects of spatial structure). Progress in fundamental laws of geography will be difficult and slow going. Nonetheless, it is an endeavor with immense benefits for sorting out the complex world we live in.

Is It ‘Deduction’ or ‘Induction’, My Dear Watson?

Sherlock Holmes often talks about ‘deductive reasoning’, but was he really using deduction or induction. Although by definition these two approaches appear to be opposites, in practice, the differences between the two can be subtle.

A simplified contrast between deductive and inductive reasoning is that deduction is reasoning from the top down and induction is reasoning from the bottom up. However, the modern definitions of these philosophies have many nuances, which address issues with both of these over-simplified descriptions and blur the lines between the two. Nonetheless, as a natural scientist, I view deduction as the formation of generalized rules that help prediction. Some have argued that deduction does not allow for a conclusion to be false, but that would not be science. It has been my observation that scientists using deductive reasoning make use of exceptions when they are discovered to refine understanding. For example, in this scene, Sherlock over-extends a generalization and learns a lesson that he probably won’t forget:

Inductive reasoning seems less comfortable with prediction, but provides better specifics about what is known and what is unknown. In other words, deduction has a problem with ‘not knowing what you don’t know’ whereas induction is more cautious by stating ‘what is known and how well it is known.’ Clearly these two philosophies have their respective strengths and can be used in concert with one another. However, it is interesting how deduction has come in and out of favor over time.

During the Age of Enlightenment (18-19th centuries), deduction was a popular mode of science. Because Sherlock (as well as the books’ author, Sir Arthur Conan Doyle) lived during the later part of this time, it makes sense that this would be the term chosen. Intriguingly, sometime between the 19th century and today, there has been a shift in scientific emphasis from the deductive to inductive approach. I suspect two potential causes for this.

First, in the 19th and early 20th centuries, science got into some hot water by making generalizations about humans. At the time, this wasn’t considered offensive and it was used improperly to support racist philosophies that were prevalent. However, today we do not tolerate racism (rightfully so!), and anything that was associated with racism has fallen out of favor (see environmental determinism).

The second reason for the shift, I believe, is due to the momentum of the ‘quantitative revolution’ in science. This transition in science is less of a revolution than it is often heralded to be. It also took various forms and happened at diverse times in different disciplines. In any case, it was a paradigm shift that put greater emphasis on the quantitative measurement of data and firmly placed a line between ideas that were supported by the data and those that were mostly speculative. Because speculative “arm waving” was fairly common in scientific literature prior to the ‘quantitative revolution’, I believe deduction suffered from its association. In addition, even though deductive approaches did regularly rely on quantitative measurements, the ‘quantitative revolution’ increased the focus on the data (fact collection). I think this is a false shift (deduction can be just as quantitative as induction), but the spirit of this ‘revolution’ brought induction into favor.

So, was Sherlock using deduction? Yes, but his methods would be questionably scientific by today’s standards. Not because of his strategy, but because the generalizations he used to make his conclusions were based on experience (perhaps we could call this anecdotal evidence), as opposed to fully tested experiments using quantitative data to prove (or disprove) the generalization. Although in all fairness, Sherlock did say, “It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts” (from A Scandal in Bohemia) as well as “Data! Data! Data! I can’t make bricks without clay” (from the Adventure of the Copper Beeches).

Here is how Sherlock described his process, “In solving a problem of this sort, the grand thing is to be able to reason backwards. That is a very useful accomplishment, and a very easy one, but people do not practice it much. In the every-day affairs of life it is more useful to reason forwards, and so the other comes to be neglected…Let me see if I can make it clearer. Most people, if you describe a train of events to them, will tell you what the result will be. They can put those events together in their minds, and argue from them that something will come to pass. There are few people, however, who, if you told them the result, would be able to evolve from their own inner consciousness what the steps were which led up to that result. This power is what I mean when I talk of reasoning backwards, or analytically.” (from A Study in Scarlet)

More quotes from Sherlock Holmes on this topic

Is It a Scientific Theory or Hypothesis?

This is a common question addressing a popular misconception about how science classifies the knowledge that it has accumulated. The levels of hypothesis to theory to law often get interpreted as classes of confidence. However, this is not really right. The missing piece here is spatial scale!

It isn’t easy to draw the line for what is scientifically known and what is not. A major reason this line is so blurry is that knowledge can be applicable at different spatial scales. The three categories of scientific understanding (hypothesis, theory, and law) are often defined in introductory science courses as describing how thoroughly a concept has been tested. Although theories do need more testing than hypothesis and laws more testing than theories, the amount of testing required is only a consequence of the real definitions. These three classes are actually describing the extent over which we know an idea to be true. The larger the extent, the more testing that will be required to know that the concept is applicable across the entire extent.

Because we encounter phenomena at a very local scale, this is usually the beginning of our ideas about how the world works. Based on these few observations, or anecdotes, we form a hypothesis about what is happening and why. In popular culture, this idea might be called a ‘theory’, but science would require proof that the concept applies in more situations before elevating it to a theory.

As we gather more data and test the idea more, we can start to explain connections between several phenomena. When we have evidence that a concept has greater applicability than to just a few anecdotes, then we can upgrade it to a theory. Although more testing was required to make this change in knowledge classification, the important part is that we now have confidence in a wider applicability of the idea.

At the top of this hierarchy of scales are scientific laws. These are concepts that have proven to be universally true. Clearly, to be confident that something is universal requires a lot of testing, which is why there is sometimes confusion about the meaning of these categories of scientific knowledge. Being universally applicable does not mean that laws cannot have defined conditions under which they apply. In fact, many are limited to certain conditions, but even with those conditions, scientific laws are expected to be reliable no matter where you go in space.

The clarifications I provide here do not contradict popular definitions, but they point out that oversimplified definitions focus on the wrong part of the scientific knowledge classification, which leads to misconceptions. At a single site, one can make millions of observations of the same thing happening the same way. However, the tested explanation for this one location will never be considered a law, despite a high confidence in that explanation being able to predict the same event occurring again. This is key to understanding why some scientific explanations with a high level of confidence will never get promoted to a higher category of scientific understanding.

To give a specific example, we have observed on the planet Earth that the acceleration due to gravity is between 9.76 and 9.84 m/s2. Despite the high level of confidence in the truth and predictability of this fact, it alone could never be promoted to a law. It cannot be considered a law because it lacks universal applicability. Observations of this phenomenon (plus celestial bodies) did lead to Newton’s Law of Universal Gravitation, which identifies a gravitational force between any two masses that is equal in magnitude for each mass, and is aligned to draw the two masses toward each other:

$F = G \left ( \frac{m_1m_2}{r^{2}} \right )$
where, m1 and m2 are the two masses, G is the gravitational constant, and r is the distance between the two masses.

This equation is capable of being a scientific law because it has been generalized to be universally applicable. Of course, rates of gravitational acceleration on Earth are still useful and reliable. The same can be said for many other pieces of scientific information that cannot be or have not yet been established for greater extents.

For more, check out: Berkeley’s webpage on “Science at multiple levels”

The Scientific Method: Does Anybody Really Use It?

It can be hard to match what we learned about the scientific method in school with the scientific literature being published. There are a variety of reasons for this, but generally it is an issue of what is practical (especially within the time frames that sources of funding expect results).

Because the classic description of the scientific method is a bit idealized, I’ll attempt to summarize the process in a way closer to reality:

1. Identify opportunities to expand knowledge. A scientist puts together an idea based on what is already known and creative ideas on how to make improvements (observation and maybe hypothesizing). The new research could help fill gaps in knowledge or attempt to improve methods. These new questions are often guided by the problems that society needs to solve.
2. Design a systematic means to collect and analyze data that will accomplish defined goals. The scientist designs a research plan that will expand knowledge, but because modern science is often chipping away at complicated issues, there may not be a formal hypothesis tested.

If a hypothesis is not tested, is the research still scientific? Yes. Maybe we should call this activity scientific exploration in contrast to scientific experimentation, but both activities are needed to expand the body of scientific knowledge. The key is how the data is collected and analyzed. In both cases, reliable (precise) measurements must be taken.

3. Do the research (experiment and assess data). In real life, things rarely go as planned. To keep the research scientific, one needs to keep track of the surprises that come up and any adaptations that were done to the work in response to them.
4. Document. This is a critical and sometimes tricky step. Nobody has time to read a detailed log on everything that happened throughout the research project. The work has to be synthesized, but in a way that is transparent enough for others to evaluate and potentially build on the work.

In the practical world of doing scientific work, it is often necessary to find projects that make incremental progress towards a bigger scientific question. Therefore, useful projects could be primarily about the collection of data or simply the improvement of a method. In this context, a critical challenge to scientific work is making valid comparisons between studies. This need for comparisons makes the use of standardized measurements and analysis methods very important.

Although, the scientific method may not be clearly visible in modern research studies, the logic principles behind it are still essential to scientific research. The causes of phenomenon (whether it be a bug in a software program or the distribution of soil properties around the world) are determined by a systematic process to eliminate alternative causes. Only when all of the alternative possibilities are eliminated can conclusions be drawn. When single studies cannot complete this task alone, they advance science more cautiously by only proposing what the results suggest and being careful to not extend conclusions beyond what is supported by available data.

What is Science?

For my first blog post, it makes sense to start with the basics. And one of the most fundamental concepts for a researcher to establish is the definition of science. As a starting point, let’s begin with the definition from the Oxford dictionary (which is very similar to definitions found elsewhere):

def. Science – the intellectual and practical activity encompassing the systematic study of the structure and behavior of the physical and natural world through observation and experiment.

I think the key phrase in this definition is that science is a “systematic study,” which establishes the epistemology of science. The critical approach for evaluating information and ideas is what separates scientific knowledge from all the other ideas that are out there. The important thing to understand is that science is very deliberate in defining what is known and what is unknown. The fact that scientific knowledge has a high standard for establishing something as ‘known’ is what makes it special. The drive to move more things from the unknown category to the known category is what makes science exhilarating.

Part of the special standard of knowledge for science is the required consideration of the possibility that an idea can be disproved. If an idea is not testable (falsifiable), then it cannot meet the standards of scientific knowledge. Similarly, science is always seeking to improve itself by never assuming that new information couldn’t provide reason for revising or even throwing out old ideas that seemed well proven in the past.

After saying all of this, I should make mention of the practical work of science. Expanding knowledge is a general goal of science, but to serve other societal needs, we often focus the use of this knowledge for prediction. In order for us as people to decide on the best actions to take, we want to know what the impacts of those decisions will be. Therefore, a lot of scientific work focuses on finding better ways to do things and identifying hazards. Of course, prediction is a risky business, but science provides us with a means for increasing our confidence in those predictions.

The philosophy of science is remarkably still heavily debated, but maybe that makes sense, because one of science’s strengths is that nothing should be left unquestioned. However, there are several central principles that modern scientists adhere to. For a more in-depth look at this topic, check out: A Guide to Understanding Science 101