In this chapter we describe a framework of density-ratio estimation by density-ratio fitting under the Bregman divergence (Bregman, 1967). This framework is a natural extension of the least-squares approach described in Chapter 6, and it includes various existing approaches as special cases (Sugiyama et al., 2011a).

In Section 7.1 we first describe the framework of density-ratio fitting under the Bregman divergence. Then, in Section 7.2, we show that various existing approaches can be accommodated in this framework, such as kernel mean matching (see Section 3.3), logistic regression (see Section 4.2), Kullback–Leibler importance estimation procedure (see Section 5.1), and least-squares importance fitting (see Section 6.1).We then show other views of the density-ratio fitting framework in Section 7.3. Furthermore, in Section 7.4, a robust density-ratio estimator is derived as an instance of the density-ratio fitting approach based on Basu's power divergence (Basu et al., 1998). The chapter is concluded in Section 7.5.

Basic Framework

A basic idea of density-ratio fitting is to fit a density ratio model r(x) to the true density-ratio function r*(x) under some divergence (Figure 7.1). At a glance, this density-ratio fitting problem may look equivalent to the regression problem, which is aimed at learning a real-valued function [see Section 1.1.1 and Figure 1.1(a)]. However, density-ratio fitting is essentially different from regression because samples of the true density-ratio function are not available. Here we employ the Bregman (BR) divergence (Bregman, 1967) for measuring the discrepancy between the true density-ratio function and the density-ratio model.