Information is a fashionable concept with many facets, among which the quantitative one–our subject–is perhaps less striking than fundamental. At the intuitive level, for our purposes, it suffices to say that information is some knowledge of predetermined type contained in certain data or pattern and wanted at some destination. Actually, this concept will not explicitly enter the mathematical theory. However, throughout the book certain functionals of random variables will be conveniently interpreted as measures of the amount of information provided by the phenomena modeled by these variables. Such information measures are characteristic tools of the analysis of optimal performance of codes, and they have turned out to be useful in other branches of mathematics as well.

Intuitive background

The mathematical discipline of information theory, created by C. E. Shannon (1948) on an engineering background, still has a special relation to communication engineering, the latter being its major field of application and the source of its problems and motivation. We believe that some familiarity with the intuitive communication background is necessary for a more than formal understanding of the theory, let alone for doing further research. The heuristics, underlying most of the material in this book, can be best explained on Shannon's idealized model of a communication system (which can also be regarded as a model of an information storage system). The important question of how far the models treated are related to, and the results obtained are relevant for, real systems will not be addressed.