The entropy

In 1948 Claude Shannon realized that there was a way to quantify the intuitive notion that some messages contain more information than others. He approached this problem by saying that a message provides information when it reveals which of a set of possibilities has occurred. This concept certainly makes sense given our intuitive notion of uncertainty: information should reduce our uncertainty about a set of possibilities. He then asked, how long must a message be to convey one of M possibilities?

To answer this question we must first specify the alphabet we are using. Specifically, we need to fix the number of letters, or symbols, in our alphabet. It is simplest to take the smallest workable alphabet, one that contains only two symbols, and we will call these 0 and 1. Imagine now that there is a new movie showing, and you are trying to decide whether to go see it. You call a friend who has seen it, Alice, and ask her whether she liked it. To answer your question yes or no she need only send you one symbol, so long as you have agreed beforehand that 1 will represent yes and 0 no. Note that two people must always agree beforehand on the meaning of the symbols they use to communicate – the only reason you can read this text is because we have a prior agreement about the meaning of each word.