In this chapter our aim is to provide an introduction to the nonlinear least squares problem. In practice many of the problems of interest are nonlinear in nature. These include several problems of interest in radar and satellite meteorology, exploration problems in geology, and tomography, to mention a few. In Section 7.1, we describe the first-order method which in many ways is a direct extension of the ideas developed in Chapters 5 and 6. This method is based on a classical idea from numerical analysis – replacing h(x) by its linear approximation at a given operating point xc, and solving a linear problem to obtain a new operating point, xnew, which is closer to the target state, x, than the original starting point. By repeatedly applying this idea, we can get as close to the target state as needed. The second-order counterpart of this idea is to replace h(x) by its quadratic approximation and, except for a few algebraic details, this method essentially follows the above iterative paradigm. This second-order method is described in Section 7.2.

A first-order method

Let x ∈ ℝn denote the state of a system under observation. Let z ∈ ℝm denote a set of observables which depend on the state x, and let z = h(x) be a representation of the physical laws that relate the underlying state to the observables – temperature to the energy radiated measured by a satellite or rain to the reflectivity measured by a doppler radar, where h(x) = (h1(x), h2(x), …, hm(x))T is a m-vector-valued function of the vector x, and hi(x) : ℝn → ℝ is the scalar valued function which is the ith component of h(x) for i = 1, 2, …, m.