This chapter will introduce concepts in statistics, probability theory, and information theory. It is not a comprehensive treatment, but will give the reader a general understanding of the principles on which the methods in this book are based.

Estimating Probability Distributions

The use of probability in daily life is often difficult. It seems that the human mind is best suited to dealing with certainties and clear outcomes for planned actions. Probability theory is used when outcomes are less certain, and many possibilities exist.

Consider the statement in a weather report: On Monday, there is a 20% chance of rain. What does that reference to 20% chance mean? Ultimately, we only have to wait for Monday, and see if it rains or not. So, this seems to be less a statement about facts than about our knowledge of the facts. In other words, it reflects our uncertainty about the facts (in this case, the future weather).

For the human mind dealing with probabilistic events creates complexities. To address a 20% chance of rain, we cannot decide to carry 20% of an umbrella. We either risk carrying unnecessary weight with us or risk getting wet. So, a typical response to this piece of information is to decide that it will not rain and ignore the less likely possibility. We do this all the time.