Book contents
- Frontmatter
- Dedication
- Contents
- Preface
- Part I Machine learning and kernel vector spaces
- Part II Dimension-reduction: PCA/KPCA and feature selection
- 3 PCA and kernel PCA
- 4 Feature selection
- Part III Unsupervised learning models for cluster analysis
- Part IV Kernel ridge regressors and variants
- Part V Support vector machines and variants
- Part VI Kernel methods for green machine learning technologies
- Part VII Kernel methods and statistical estimation theory
- Part VIII Appendices
- References
- Index
3 - PCA and kernel PCA
from Part II - Dimension-reduction: PCA/KPCA and feature selection
Published online by Cambridge University Press: 05 July 2014
- Frontmatter
- Dedication
- Contents
- Preface
- Part I Machine learning and kernel vector spaces
- Part II Dimension-reduction: PCA/KPCA and feature selection
- 3 PCA and kernel PCA
- 4 Feature selection
- Part III Unsupervised learning models for cluster analysis
- Part IV Kernel ridge regressors and variants
- Part V Support vector machines and variants
- Part VI Kernel methods for green machine learning technologies
- Part VII Kernel methods and statistical estimation theory
- Part VIII Appendices
- References
- Index
Summary
Introduction
Two primary techniques for dimension-reducing feature extraction are subspace projection and feature selection. This chapter will explore the key subspace projection approaches, i.e. PCA and KPCA.
(i) Section 3.2 provides motivations for dimension reduction by pointing out (1) the potential adverse effect of large feature dimensions and (2) the potential advantage of focusing on a good set of highly selective representations.
(ii) Section 3.3 introduces subspace projection approaches to feature-dimension reduction. It shows that the well-known PCA offers the optimal solution under two information-preserving criteria: least-squares error and maximum entropy.
(iii) Section 3.4 discusses several numerical methods commonly adopted for computation of PCA, including singular value decomposition (on the data matrix), spectral decomposition (on the scatter matrix), and spectral decomposition (on the kernel matrix).
(iv) Section 3.5 shows that spectral factorization of the kernel matrix leads to both kernel-based spectral space and kernel PCA (KPCA) [238]. In fact, KPCA is synonymous with the kernel-induced spectral feature vector. We shall show that nonlinear KPCA offers an enhanced capability in handling complex data analysis. By use of examples, it will be demonstrated that nonlinear kernels offer greater visualization flexibility in unsupervised learning and higher discriminating power in supervised learning.
Why dimension reduction?
In many real-world applications, the feature dimension (i.e. the number of features or attributes in an input vector) could easily be as high as tens of thousands. Such an extreme dimensionality could be very detrimental to data analysis and processing.
- Type
- Chapter
- Information
- Kernel Methods and Machine Learning , pp. 79 - 117Publisher: Cambridge University PressPrint publication year: 2014