In the preceding chapters we investigated in detail the scenario of a student perceptron learning from a teacher perceptron. This is a typical example of what is commonly referred to as supervised learning. But we all gratefully acknowledge that learning from examples does not always require the presence of a teacher!

However, what is it that can be learned besides some specific classification of examples provided by a teacher? The key observation is that learning from unclassified examples is possible if their distribution has some underlying structure. The main issue in unsupervised learning is then to extract these intrinsic features from a set of examples alone. This problem is central to many pattern recognition and data compression tasks with a variety of important applications [110].

Far from attempting to review the many existing approaches to unsupervised learning, we will show in the present chapter how statistical mechanics methods introduced before can be applied to some special scenarios of unsupervised learning closely related to the teacher–student perceptron problem. This will illustrate on the one hand how statistical mechanics can be used for the analysis of unsupervised situations, while on the other hand we will gain new understanding of the supervised problem by reformulating it as a special case of an unsupervised one.