Founded by Shannon, information theory deals with the fundamental principles of communication. The two most important questions answered by this theory are how much we can compress a given data source and how much data we can transmit in a given communication channel.

Information theory is essentially statistically minded. Data sources are modeled as random processes, and transmission channels are also modeled in probabilistic terms. The theory does not deal with the content of information – it deals with the frequency at which symbols (letters, figures, etc.) appear or are processed but not their meaning. A statistical model is not the only option. Non-statistical theories also exist (e.g., Kolmogorov complexity). However, in this section and throughout this book, we will only use the statistical tools.

Information theory is of central importance in quantum cryptography. It may be used to model the transmission of the key elements from Alice to Bob. Note that what may happen on the quantum channel is better described using quantum information theory – see Chapter 4. Yet, the key elements chosen by Alice and those obtained by Bob after measurement are classical values so, for instance, the transmission errors can accurately be modeled using classical information theory. Reconciliation, in particular, requires classical information-theoretic techniques.

Source coding

Source coding is the first problem that information theory addresses. Assume that a source emits symbols xi from an alphabet χ and that it can be modeled by the random variable X on χ. For instance, the source can be the temperature measured by some meteorological station at regular intervals or the traffic on a network connection.