2. Definition of the control model $\mathcal{M}$

Elements of the control model $\mathcal{M}$ .

The MDP under consideration consists of the following elements:

• The state space $\textbf{X}$ and the action space $\textbf{A}$ are both Borel spaces endowed with their respective Borel $\sigma$ -algebras $\boldsymbol{\mathfrak{B}}(\textbf{X})$ and $\boldsymbol{\mathfrak{B}}(\textbf{A})$ .

• For each $x\in\textbf{X}$ , the nonempty measurable set $\textbf{A}(x)\subseteq \textbf{A}$ stands for the set of available actions in state x. Let $\textbf{K}=\{(x,a)\in\textbf{X}\times\textbf{A}\,:\, a\in \textbf{A}(x)\}$ be the family of feasible state–action pairs. Assumption (A.1) below will ensure that $\textbf{K}\in\boldsymbol{\mathfrak{B}}(\textbf{X})\otimes\boldsymbol{\mathfrak{B}}(\textbf{A})$ .

• The initial distribution of the system is the probability measure $\nu\in\boldsymbol{\mathcal{P}}(\textbf{X})$ , and the transitions of the system are given by a stochastic kernel Q on $\textbf{X}$ given $\textbf{K}$ .

• Finally, regarding the performance criterion to be optimized (which we will define later), a nontrivial measurable subset of the state space $G\subseteq\textbf{X}$ is given.

Our main conditions on the control model $\mathcal{M}$ are stated next. In Assumption (A.2) we introduce a reference probability measure $\lambda\in\boldsymbol{\mathcal{P}}(\textbf{X})$ .

Control policies. The space of admissible histories up to time n for the controlled process is denoted by $\textbf{H}_{n}$ for $n\in\mathbb{N}$ . It is defined recursively by $\textbf{H}_0=\textbf{X}$ and $\textbf{H}_n=\textbf{K}\times\textbf{H}_{n-1}$ for any $n\ge1$ , and it is endowed with the corresponding product $\sigma$ -algebra. A control policy $\pi$ is a sequence $\{\pi_n\}_{n\ge0}$ of stochastic kernels on $\textbf{A}$ given $\textbf{H}_n$ , denoted by $\pi_n(da|h_n)$ , such that for each $n\ge0$ and $h_n=(x_0,a_0,\ldots,x_n)\in\textbf{H}_n$ we have $\pi_n(\textbf{A}(x_n)|h_n)=1$ . The set of all policies is denoted by $\boldsymbol{\Pi}$ .

Dynamics of the control model. The canonical sample space of all possible state and actions is $\boldsymbol{\Omega}=(\textbf{X}\times\textbf{A})^\infty$ , and it is endowed with the product $\sigma$ -algebra $\boldsymbol{\mathcal{F}}$ . Thus, a generic element $\omega\in\boldsymbol{\Omega}$ consists of a sequence of the form $(x_0,a_0,x_1,a_1,\ldots)$ , where $x_{n}\in \textbf{X}$ and $a_{n}\in\textbf{A}$ for any $n\in\mathbb{N}$ . For $n\in\mathbb{N}$ , the projection functions $X_{n}$ and $A_{n}$ from $\boldsymbol{\Omega}$ to $\textbf{X}$ and $\textbf{A}$ respectively are defined by $X_{n}(\omega)= x_{n}$ and $A_{n}(\omega)= a_{n}$ for $\omega=(x_0,a_0,\ldots,x_n,a_n,\ldots)$ , and they are called the state and control variables at time n. The history process up to time n is denoted by $H_n=(X_0,A_0,\ldots,X_n)$ .

For any policy $\pi\in\boldsymbol{\Pi}$ , there exists a unique probability measure $\mathbb{P}^{\pi}_{\nu}$ on $(\boldsymbol{\Omega},\boldsymbol{\mathcal{F}})$ supported on $\textbf{K}^\infty$ (that is, $\mathbb{P}^{\pi}_{\nu}( \textbf{K}^\infty)=1)$ which satisfies the following conditions:

1. (i) We have $\mathbb{P}^{\pi}_{\nu} \{X_0\in D\}=\nu(D)$ for any $D\in \boldsymbol{\mathfrak{B}}(\textbf{X})$ .

2. (ii) For any $n\ge0$ and $C\in\boldsymbol{\mathfrak{B}}(\textbf{A})$ we have $\mathbb{P}^{\pi}_{\nu}\{ A_n\in C | H_n\} = \pi_n(C|H_n)$ .

3. (iii) For any $n\ge0$ and $D\in\boldsymbol{\mathfrak{B}}(\textbf{X})$ we have $\mathbb{P}^{\pi}_{\nu}\{ X_{n+1}\in D| H_n,A_n\}=Q(D | X_n,A_n)$ .

The probability measure $\mathbb{P}^{\pi}_{\nu}$ is usually referred to as a strategic probability measure. The corresponding expectation operator is $\mathbb{E}^{\pi}_{\nu}$ . The set of all strategic probability measures $\{\mathbb{P}^{\pi}_{\nu}\}_{\pi\in\boldsymbol{\Pi}}$ is denoted by $\boldsymbol{\mathcal{S}}_{\nu}\subseteq\boldsymbol{\mathcal{P}}(\boldsymbol{\Omega})$ .

We will also denote by $\mathbb{P}^\pi_{\nu,n}$ the distribution of the random variable $H_n$ ; more formally, for any $n\ge0$ we define $\mathbb{P}^\pi_{\nu,n}$ as the pushforward probability measure on $(\textbf{X}\times\textbf{A})^n\times\textbf{X}$ given by $\mathbb{P}^\pi_{\nu}\circ H_n^{-1}$ . Consequently, we have $\mathbb{P}^\pi_{\nu,n}(\textbf{H}_n)=1$ ; note also that $\mathbb{P}^\pi_{\nu,0}=\nu$ .

Optimality criterion. The performance criterion that we want to maximize is the probability of visiting the set G infinitely often. Let us denote by $N_{G}\,:\,\boldsymbol{\Omega}\rightarrow\mathbb{N}\cup\{\infty\}$ the integer-valued process that counts the total number of visits of the state process to the set G; that is,

\begin{align*}N_G(\omega)=\sum_{n=0}^\infty \textbf{I}_{G}(X_n(\omega)).\end{align*}

For notational convenience, if $I\subseteq\mathbb{N}$ is a set of decision epochs, we let $N_G(I)=\sum_{n\in I} \textbf{I}_{G}(X_n)$ , which stands for the number of visits to G at times in I.

For the initial distribution $\nu$ , the value function of the control model $\mathcal{M}$ is given by

(2.2) \begin{equation}V^*(\nu)=\sup_{\pi\in\boldsymbol{\Pi}} \mathbb{P}^{\pi}_{\nu} \{N_G=\infty\}.\end{equation}

A policy attaining the above supremum is said to be optimal. In the sequel, the model $\mathcal{M}$ introduced in this section will be referred to as the primary or original control problem.

Stationary Markov policies. Let $\boldsymbol{\Pi}_{s}$ be the set of stochastic kernels $\pi$ on $\textbf{A}$ given $\textbf{X}$ such that $\pi(\textbf{A}(x)|x)=1$ for each $x\in\textbf{X}$ . We have that $\boldsymbol{\Pi}_{s}$ is nonempty as a consequence of our hypotheses. Indeed, from the Kuratowski–Ryll-Nardzewski selection theorem [Reference Aliprantis and Border1, Theorem 18.13] we derive the existence of a measurable selector for the correspondence $x\mapsto\textbf{A}(x)$ ; that is, there exists a measurable $f\,:\,\textbf{X}\rightarrow\textbf{A}$ such that $f(x)\in\textbf{A}(x)$ for every $x\in\textbf{X}$ . Then we have that $\pi(da|x)=\delta_{f(x)}(da)$ indeed belongs to $\boldsymbol{\Pi}_{s}$ .

We say that $\{\pi_n\}_{n\in\mathbb{N}}\in\boldsymbol{\Pi}$ is a stationary Markov policy if there is some $\pi\in\boldsymbol{\Pi}_{s}$ such that

\begin{align*}\pi_n(\cdot|x_0,a_0,b_0,\ldots,x_n)=\pi(\cdot|x_{n})\quad\hbox{for all $n\ge0$ and $h_n=(x_0,a_0,\ldots,x_n)\in\textbf{H}_n$}.\end{align*}

Without risk of confusion, we will identify the set of all stationary Markov policies with $\boldsymbol{\Pi}_{s}$ itself. The set of all strategic probability measures for the stationary Markov policies $\{\mathbb{P}^{\pi}_{\nu}\}_{\pi\in\boldsymbol{\Pi}_s}$ is denoted by $\boldsymbol{\mathcal{S}}^s_{\nu}$ .

Proof. Suppose that $Y\in\boldsymbol{\mathfrak{B}}(\textbf{X})$ with $\lambda(Y)=1$ is such that $\pi(da|x)=\pi'(da|x)$ for every $x\in Y$ . We will prove by induction on $n\ge0$ that $(X_0,A_0,\ldots,X_n,A_n)$ has the same distribution under $\mathbb{P}^{\pi}_{\nu}$ and under $\mathbb{P}^{\pi'}_{\nu}$ . Letting $n=0$ , and given $D\in\boldsymbol{\mathfrak{B}}(\textbf{X})$ and $C\in\boldsymbol{\mathfrak{B}}(\textbf{A})$ , we have

\begin{eqnarray*}\mathbb{P}^\pi_{\nu}\{X_0\in D,A_0\in C\} &=& \, \int_D \pi(C|x)\frac{d\nu}{d\lambda}(x)\, \lambda(dx)\ =\ \int_{D\cap Y} \pi(C|x)\frac{d\nu}{d\lambda}(x)\, \lambda(dx) \\&=& \, \int_{D} \pi'(C|x)\frac{d\nu}{d\lambda}(x)\, \lambda(dx) \ =\ \mathbb{P}^{\pi'}_{\nu}\{X_0\in D,A_0\in C\}.\end{eqnarray*}

Assuming the result is true for $n\ge0$ , and again letting $D\in\boldsymbol{\mathfrak{B}}(\textbf{X})$ and $C\in\boldsymbol{\mathfrak{B}}(\textbf{A})$ , we have

\begin{align*}\mathbb{P}^\pi_{\nu}\{ X_{n+1}\in D, A_{n+1}\in C\mid X_n=x,A_n=a\}= \int_D \pi(C|y) q(x,a,y)\lambda(dy), \end{align*}

which, by the same argument, is equal to $\mathbb{P}^{\pi'}_{\nu}\{ X_{n+1}\in D, A_{n+1}\in C\mid X_n=x,A_n=a\}$ . The conditional distributions and the distribution of $(X_n,A_n)$ being the same under $\pi$ and $\pi'$ , we conclude the result.

Young measures. Let $\boldsymbol{\mathcal{R}}\supseteq\boldsymbol{\Pi}_s$ be the set of stochastic kernels $\gamma$ on $\textbf{A}$ given $\textbf{X}$ such that $\gamma(\textbf{A}(x)|x)=1$ for $\lambda$ -almost every $x\in\textbf{X}$ . On $\boldsymbol{\mathcal{R}}$ we consider the equivalence relation $\approx$ defined as $\gamma\approx\gamma'$ when $\gamma(\cdot|x)=\gamma'(\cdot|x)$ for $\lambda$ -almost all $x\in\textbf{X}$ . The corresponding quotient space $\boldsymbol{\mathcal{R}}\,/\approx$ , denoted by $\boldsymbol{\mathcal{Y}}$ , is the so-called set of Young measures. As a consequence of Lemma 2.1 and Remark 2.2, there is no loss of generality in identifying the set of stationary Markov policies $\boldsymbol{\Pi}_s$ with $\boldsymbol{\mathcal{Y}}$ , since all the elements in the same $\approx$ -equivalence class have the same strategic probability measure. Thus we will indiscriminately use the symbols $\boldsymbol{\Pi}_s$ and $\boldsymbol{\mathcal{Y}}$ to refer to both the stationary Markov policies and the set of Young measures. Hence, we will write

\begin{align*}\boldsymbol{\mathcal{S}}^s_{\nu}=\{\mathbb{P}^\pi_{\nu}\}_{\pi\in\boldsymbol{\mathcal{Y}}}\subseteq\boldsymbol{\mathcal{S}}_{\nu}.\end{align*}

The set of Young measures $\boldsymbol{\mathcal{Y}}$ will be equipped with the narrow (stable) topology. This is the coarsest topology which makes the following mappings continuous:

\begin{align*}\pi\mapsto \int_{\textbf{X}}\int_{\textbf{A}}f(x,a) \pi (da|x)\lambda(dx),\end{align*}

for any $f\in {\textit{Car}}(\textbf{X}\times\textbf{A})$ such that for some $\Phi$ in $L^1(\textbf{X},\boldsymbol{\mathfrak{B}}(\textbf{X}),\lambda)$ we have $|f(x,a)| \leq \Phi(x)$ for every $(x,a)\in\textbf{X}\times\textbf{A}$ ; see [Reference Balder2, Theorem 2.2]. By [Reference Balder4, Lemma 1], the set $\boldsymbol{\mathcal{Y}}$ endowed with the narrow topology becomes a compact metric space. We state this result in the next lemma.

The $ws^\infty$ -topology. Finally, we introduce the so-called $ws^\infty$ -topology on $\boldsymbol{\mathcal{S}}_{\nu}$ . Let

\begin{align*}\boldsymbol{\mathcal{U}}_{0}=\boldsymbol{\mathcal{B}}(\textbf{X})\quad\hbox{and}\quad \boldsymbol{\mathcal{U}}_{n}={\textit{Car}}_b(\textbf{X}^{n+1}\times\textbf{A}^n)\quad\hbox{for $n\ge1$},\end{align*}

where the arguments of $g\in\boldsymbol{\mathcal{U}}_n$ will be sorted as $(x_0,a_0,\ldots,x_{n-1},a_{n-1},x_{n})$ . The $ws^\infty$ -topology on the set $\boldsymbol{\mathcal{S}}_{\nu}$ —see [Reference Schäl28]—is the smallest topology that makes the mappings

\begin{align*}\mathbb{P} \mapsto \int_{\boldsymbol{\Omega}} g(H_n) \textrm{d} \mathbb{P} \end{align*}

continuous for any $n\ge 0$ and $g\in\boldsymbol{\mathcal{U}}_{n}$ . Therefore, a sequence $\{\mathbb{P}_k\}_{k\in\mathbb{N}}\subseteq\boldsymbol{\mathcal{S}}_{\nu}$ converges to $\mathbb{P}\in\boldsymbol{\mathcal{S}}_{\nu}$ for the $ws^\infty$ -topology, denoted by $\mathbb{P}_k\Rightarrow \mathbb{P}$ , if and only if

(2.3) \begin{equation}\lim_{k\rightarrow\infty} \int_{\boldsymbol{\Omega}} g(H_n) \textrm{d} \mathbb{P}_k=\int_{\boldsymbol{\Omega}} g(H_n) \textrm{d} \mathbb{P}\end{equation}

for any $n\in\mathbb{N}$ and $g\in\boldsymbol{\mathcal{U}}_{n}$ .

The proof of Lemmas 2.3 and Theorem 2.1 below are fairly involved; they are presented in Appendix A. Note that Lemma 2.3 simply states that the mapping that associates to a Young measure its corresponding strategic probability measure is continuous.

It is worth mentioning that compactness of $\boldsymbol{\mathcal{S}}_{\nu}$ is a known result (see [Reference Balder3] or [Reference Schäl28]), whereas the new result that we establish here is compactness of $\boldsymbol{\mathcal{S}}^s_{\nu}$ .

3. The control model $\mathcal{M}_{\beta}$

By its definition, the performance criterion introduced above is of asymptotic type, and it cannot be written in an additive form. Therefore, it appears difficult to study this type of control problem directly by using the traditional techniques, such as the dynamic programming or linear programming approach. To address this difficulty, we introduced in [Reference Dufour and Prieto-Rumeau15], in the particular context of a countable state space, an auxiliary model $\mathcal{M}_{\beta}$ as a tool for analyzing the original control problem.

Elements of the control model $\mathcal{M}_{\beta}$ . Given $0<\beta<1$ , the auxiliary control model $\mathcal{M}_{\beta}$ , which is based on the primary control model $\mathcal{M}$ , is a ‘total-expected-reward’ MDP defined as follows:

• The state and action spaces are augmented with isolated points,

  \begin{align*}\hat{\textbf{X}}=\textbf{X}\cup\{\Delta\}\quad\hbox{and}\quad \hat{\textbf{A}}=\textbf{A}\cup\{a_{\Delta}\},\end{align*}
  so that $\hat{\textbf{X}}$ and $\hat{\textbf{A}}$ are again Borel spaces.

• The set of available actions when the state variable is $x\in\hat{\textbf{X}}$ is the set $\hat{\textbf{A}}(x)\subseteq\hat{\textbf{A}}$ , where

  \begin{align*}\hat{\textbf{A}}(x)=\textbf{A}(x)\ \ \hbox{for $x\in\textbf{X}$}\quad\hbox{and}\quad\hat{\textbf{A}}(\Delta)=\{a_{\Delta}\}.\end{align*}
  Let $\hat{\textbf{K}}\subseteq\hat{\textbf{X}}\times\hat{\textbf{A}}$ be the set of feasible state–action pairs; hence $\hat{\textbf{K}}=\textbf{K}\cup\{(\Delta,a_{\Delta})\}$ .

• The dynamics of this model is governed by the stochastic kernel $Q_{\beta}$ defined by

  \begin{align*}Q_{\beta}(\Gamma | x,a)=\ & \textbf{I}_{G}(x) \big[ \beta Q(\Gamma\cap\textbf{X} | x,a) + (1-\beta)\delta_{\Delta}(\Gamma) \big] \\ &+ \textbf{I}_{\textbf{X}-G}(x) Q(\Gamma\cap\textbf{X} | x,a) + \textbf{I}_{\{\Delta\}}(x) \delta_{\Delta}(\Gamma)\end{align*}
  for any $\Gamma\in \boldsymbol{\mathfrak{B}}(\hat{\textbf{X}})$ and $(x,a)\in\hat{\textbf{K}}$ .

  The initial probability distribution is the same as for $\mathcal{M}$ , that is, $\nu\in\boldsymbol{\mathcal{P}}(\textbf{X})$ . With a slight abuse of notation, we will also denote by $\nu$ the measure $\nu(\cdot\cap\textbf{X})$ on $\hat{\textbf{X}}$ .

• The reward function is $\textbf{I}_{G}(x)$ for $(x,a)\in\hat{\textbf{K}}$ , where $G\subseteq\textbf{X}$ is the measurable set given in the definition of $\mathcal{M}$ .

Hence, the model $\mathcal{M}_{\beta}$ is indeed a standard total-expected-reward MDP parametrized by $0<\beta< 1$ . The transitions generated by $Q_{\beta}$ can be interpreted very simply as follows. If the state process is in $\textbf{X}-G$ , then the transition is made according to Q, the stochastic kernel of the original model. If the state process is in G, then either the transition is made according to the original model $\mathcal{M}$ (with probability $\beta$ ), or the state process moves to $\Delta$ (with probability $1-\beta$ ). Finally, it is easy to see that $\Delta$ is an absorbing state under $a_{\Delta}$ , the unique action available at $\Delta$ . However, it should be observed that $\mathcal{M}_{\beta}$ is not necessarily an absorbing MDP in the terminology of [Reference Feinberg and Rothblum17, Section 2], because absorption in $\Delta$ may not occur in a finite expected time.

The definition of $\mathcal{M}_{\beta}$ is somewhat inspired by the well-known equivalent formulation of a discounted MDP, with discount factor $\beta$ , as a total-reward MDP in which the transition probabilities are multiplied by the discount factor $\beta$ , while $1-\beta$ is the probability of killing the process at each transition; see, e.g., [Reference Feinberg and Rothblum17, p. 132]. In our case, however, the $\beta$ factor is incorporated only when the process is in G.

Control policies. The construction and definition of the control policies for $\mathcal{M}_{\beta}$ is very similar to what was presented for the original model in Section 2. Therefore, we will just present the key elements and will skip some details. Letting $\hat{\textbf{H}}_0=\hat{\textbf{X}}$ , for $n\ge1$ we have

\begin{align*}\hat{\textbf{H}}_n=\textbf{H}_n\cup\bigcup_{j=0}^n\big( \textbf{K}^{j}\times \{ (\Delta,a_{\Delta})\}^{n-j}\times\{\Delta\}\big),\end{align*}

which includes the histories of the original control model, plus those paths reaching $\Delta$ at time $0\le j\le n$ and remaining in $\Delta$ thereafter. The set of all admissible histories is given by

(3.1) \begin{equation}\textbf{K}^{\infty}\cup\bigcup_{n=0}^\infty\big( \textbf{K}^{n}\times \{ (\Delta,a_{\Delta})\}^{\infty}\big).\end{equation}

The family of control policies for $\mathcal{M}_{\beta}$ is denoted by $\hat{\boldsymbol{\Pi}}$ , and the family of stationary Markov policies is $\hat{\boldsymbol{\Pi}}_{s}$ . These are defined similarly to those of $\mathcal{M}$ , where we replace $\textbf{X}$ , $\textbf{A}$ , and $\textbf{H}_{n}$ with $\hat{\textbf{X}}$ , $\hat{\textbf{A}}$ , and $\hat{\textbf{H}}_{n}$ , respectively.

Dynamics of the control model. Let $\hat{\boldsymbol{\Omega}}=(\hat{\textbf{X}}\times\hat{\textbf{A}})^\infty$ be the state–action canonical sample space endowed with its product $\sigma$ -algebra $\hat{\boldsymbol{\mathcal{F}}}$ . The state, action, and history process projections are denoted by $\hat{X}_n$ , $\hat{A}_n$ , and $\hat{H}_n$ , and they are defined as in Section 2. The counting processes associated to the visits of the state process to the set G are $\hat{N}_G=\sum_{n=0}^\infty \textbf{I}_{G}(\hat{X}_n)$ and $\hat{N}_G(I)=\sum_{n\in I} \textbf{I}_{G}(\hat{X}_k)$ for $I\subseteq\mathbb{N}$ .

Recall that we were considering a reference probability measure $\lambda\in\boldsymbol{\mathcal{P}}(\textbf{X})$ for the control model $\mathcal{M}$ . For the auxiliary control model, we will use the reference probability measure $\hat\lambda\in\boldsymbol{\mathcal{P}}(\hat{\textbf{X}})$ defined by

\begin{align*}\hat{\lambda}(\Gamma)=\frac{1}{2}\big(\lambda(\Gamma\cap\textbf{X})+\delta_{\Delta}(\Gamma)\big)\quad\hbox{for $\Gamma\in\boldsymbol{\mathfrak{B}}(\hat{\textbf{X}})$}.\end{align*}

Given $\hat{\pi}\in\hat{\boldsymbol{\Pi}}$ and the initial distribution $\nu$ , there exists a unique probability measure $\mathbb{P}^{\beta,\hat{\pi}}_{\nu}$ on $(\hat{\boldsymbol{\Omega}},\hat{\boldsymbol{\mathcal{F}}})$ supported on the set (3.1) of histories that models the dynamics of $\mathcal{M}_{\beta}$ . The corresponding expectation operator is denoted by $ \mathbb{E}^{\beta,\hat{\pi}}_{\nu}$ .

Optimality criterion. For the model $\mathcal{M}_{\beta}$ , the objective is the maximization of the total-expected-reward criterion given by the expected number of visits to the set G: for any $\hat{\pi}\in\hat{\boldsymbol{\Pi}}$ ,

\begin{align*} \mathbb{E}^{\beta,\hat{\pi}}_{\nu} \Big[\sum_{n=0}^\infty \textbf{I}_{G}(\hat{X}_n) \Big]= \mathbb{E}^{\beta,\hat{\pi}}_{\nu} [\hat{N}_G],\end{align*}

and we let

\begin{align*}V^*_{\beta}(\nu)=\sup_{\hat{\pi}\in\hat{\boldsymbol{\Pi}}} \mathbb{E}^{\beta,\hat{\pi}}_{\nu} [\hat{N}_G]\end{align*}

be the value function. A policy $\pi^*\in\hat{\boldsymbol{\Pi}}$ satisfying $\mathbb{E}^{\beta,\pi^{*}}_{\nu} [\hat{N}_G]=V^*_{\beta}(\nu)$ is said to be optimal.

Hence, there is a bijection between $\hat{\boldsymbol{\Pi}}$ and $\boldsymbol{\Pi}$ . Note that this establishes a bijection between $\boldsymbol{\Pi}_s$ and $\hat{\boldsymbol{\Pi}}_{s}$ as well.

For a complete overview of techniques for solving total-expected-reward MDPs, we refer to [Reference Hernández-Lerma and Lasserre19, Chapter 9]. In that reference, the value iteration algorithm, the policy iteration algorithm, and the linear programming approach are discussed. Other related results on value iteration are described in Theorems 3.5 and 3.7 in [Reference Dufour and Prieto-Rumeau15], and the linear programming (or convex analytic) approach is also studied in [Reference Dufour, Horiguchi and Piunovskiy13, Reference Dufour and Piunovskiy14].

The relation between $\mathcal{M}$ and $\mathcal{M}_{\beta}$ . Next we state a result that allows us to establish a correspondence between the performance functions of the two models $\mathcal{M}$ and $\mathcal{M}_{\beta}$ . Roughly, this will imply that for any pair of control policies on correspondence (recall Remark 3.1), the performance criterion of the model $\mathcal{M}_{\beta}$ multiplied by $1-\beta$ will converge, as the parameter $\beta$ tends towards 1, to the performance criterion of the original model $\mathcal{M}$ . This very important fact establishes the link from $\mathcal{M}$ to a criterion of an additive nature, for which it will be possible to use standard methods of analysis. In our paper, we use the so-called direct approach, along the lines developed in [Reference Balder3, Reference Balder5, Reference Nowak23, Reference Schäl28, Reference Schäl29]; roughly speaking, this consists in finding a suitable topology on the set of strategic probability measures to ensure that it is compact and that the reward functional is semicontinuous.

The next result is proved in [Reference Dufour and Prieto-Rumeau15] in the context of a countable state space. It can easily be generalized to the framework of our paper, namely, a general state space of Borel type. In order to avoid unnecessary repetition we will omit the proof here; we refer the reader to Section 3 of [Reference Dufour and Prieto-Rumeau15]. In the sequel we make the convention that $\beta^\infty=0$ for $\beta\in(0, 1)$ .

Proposition 3.1. For every $0<\beta<1$ , $\pi\in\boldsymbol{\Pi}$ , and $n\in\mathbb{N}$ we have

\begin{align*}\frac{1}{1-\beta}\cdot \mathbb{E}^{\pi}_{\nu} [1-\beta^{N_G([0,n])}]=\mathbb{E}^{\beta,\pi^{\Delta}}_{\nu}[\hat{N}_G([0,n])]\quad and \quad\frac{1}{1-\beta}\cdot \mathbb{E}^{\pi}_{\nu} [1-\beta^{N_G}]=\mathbb{E}^{\beta,\pi^{\Delta}}_{\nu} [\hat{N}_G].\end{align*}

Assumptions on $\mathcal{M}_{\beta}$ . We define the time of entrance of the state process $\{\hat{X}_n\}_{n\in\mathbb{N}}$ into a subset $\mathcal{C}\in\boldsymbol{\mathfrak{B}}(\hat{\textbf{X}})$ of the state space as the random variable $\tau_{\mathcal{C}}$ defined on $\hat{\boldsymbol{\Omega}}$ by

\begin{align*}\tau_{\mathcal{C}}(\omega)=\min\{n\in\mathbb{N}\,:\, \hat{X}_n(\omega)\in\mathcal{C}\},\end{align*}

with the usual convention that the minimum over the empty set is $\infty$ .

For a stationary Markov policy $\pi\in\boldsymbol{\Pi}_s$ , we define the stochastic kernel $Q_{\pi}$ on $\textbf{X}$ given $\textbf{X}$ as

\begin{align*}Q_{\pi}(\Gamma|x) = \int_{\textbf{X}} Q(\Gamma|x,a)\pi(da|x)\quad\hbox{for $x\in\textbf{X}$ and $\Gamma\in\boldsymbol{\mathfrak{B}}(\textbf{X})$}.\end{align*}

For $n\ge1$ , we denote by $Q^n_{\pi}$ the nth composition of $Q_{\pi}$ with itself, and we make the convention that $Q^0_{\pi}(\cdot|x)=\delta_x(\cdot)$ . Similarly, for a stationary Markov policy $\hat{\pi}\in\hat{\boldsymbol{\Pi}}_s$ of the auxiliary control model, we define the stochastic kernel $Q_{\beta,\hat{\pi}}$ on $\hat{\textbf{X}}$ given $\hat{\textbf{X}}$ and $Q^n_{\beta,\hat{\pi}}$ for $n\ge0$ .

Assumption B. For each $0<\beta<1$ and $\hat{\pi}\in\hat{\boldsymbol{\Pi}}_{s}$ , there exists a measurable set $\mathcal{C}^{\beta,\hat{\pi}}\subseteq \hat{\textbf{X}}$ satisfying $\hat{\lambda}(G\cap \mathcal{C}^{\beta,\hat{\pi}})=0$ and $Q_{\beta,\hat{\pi}}(\mathcal{C}^{\beta,\hat{\pi}} |x)=1$ for $\hat{\lambda}$ -almost all $x\in\mathcal{C}^{\beta,\hat{\pi}}$ . In addition, there exists a finite constant $K_{\beta}$ such that for every $\hat\pi\in\hat{\boldsymbol{\Pi}}_s$ ,

(3.2) \begin{align}\mathbb{E}^{\beta,\hat{\pi}}_{x}[\tau_{\mathcal{C}^{\beta,\hat{\pi}}}] \leq K_{\beta}\quad\hbox{for $\hat{\lambda}$-almost all $x\in\hat{\textbf{X}}-\mathcal{C}^{\beta,\hat{\pi}}$.} \end{align}

Observe that, under Assumption B, the control model $\mathcal{M}_{\beta}$ is not necessarily an absorbing MDP in the terminology of [Reference Feinberg and Rothblum17, Section 2], since the set $\mathcal{C}^{\beta,\hat{\pi}}$ depends on the control policy. The stability Assumption B can be informally and very loosely explained as follows. Given any fixed $0<\beta<1$ , for each stationary Markov policy $\hat\pi\in\hat{\boldsymbol{\Pi}}_s$ there exists a set $\mathcal{C}^{\beta,\hat{\pi}}$ which is closed (not in the topological sense, but in the terminology of Markov chains) and disjoint from G, and which is reached in a bounded expected time when starting from outside it (formally, all these statements hold almost surely with respect to the reference measure $\hat\lambda$ ).

We now state some consequences of Assumption B.

Proof. Indeed, if $\Gamma\in\boldsymbol{\mathfrak{B}}(\hat{\textbf{X}})$ is such that $\hat\lambda(\Gamma)=0$ , then necessarily $\Gamma\subseteq\textbf{X}$ with $\lambda(\Gamma)=0$ . By definition of the stochastic kernel $Q_{\beta}$ , and recalling (2.1), it is then clear that $Q_{\beta,\hat\pi}(\Gamma|x)=0$ for every $x\in\hat{\textbf{X}}$ . Now, supposing again that $\Gamma\in\boldsymbol{\mathfrak{B}}(\hat{\textbf{X}})$ is such that $\hat\lambda(\Gamma)=0$ , and letting $n\ge1$ and $x\in\textbf{X}$ , we have

\begin{align*}Q_{\beta,\hat\pi}^{n+1}(\Gamma|x)=\int_{\hat{\textbf{X}}} Q_{\beta,\hat\pi}(\Gamma|y) Q^n_{\beta,\hat\pi}(dy|x),\end{align*}

which indeed equals 0 because $Q_{\beta,\hat\pi}(\Gamma|y)=0$ .

The second statement is a straightforward consequence of the above results except for the case $n=0$ . In that case, however, the distribution of $\hat{X}_0$ is $\nu\ll\lambda$ .

The fact that $\mathcal{C}^{\beta,\hat\pi}$ is closed $\hat\lambda$ -almost surely under $\hat\pi$ is extended to further transitions.

Proof. Let us prove the result by induction. By Assumption B, it holds for $n=1$ . Now assume that the claim holds for some $n\ge1$ . Let $C_n\subseteq \mathcal{C}^{\beta,\hat\pi}$ , with $\hat\lambda(\mathcal{C}^{\beta,\hat\pi}-C_n)=0$ , be the set of $x\in\hat{\textbf{X}}$ for which $Q_{\beta,\hat\pi}(\mathcal{C}^{\beta,\hat\pi}|x)=1$ and $Q^n_{\beta,\hat\pi}(\mathcal{C}^{\beta,\hat\pi}|x)=1$ . If $x\in C_n$ we have

\begin{align*}Q^{n+1}_{\beta,\hat{\pi}}(\mathcal{C}^{\beta,\hat{\pi}}|x)\ge\int_{C_n} Q^{n}_{\beta,\hat{\pi}}(\mathcal{C}^{\beta,\hat{\pi}}|y) Q_{\beta,\hat{\pi}}(dy|x)=Q_{\beta,\hat{\pi}}(C_n|x).\end{align*}

Since $Q_{\beta,\hat\pi}(\cdot|x)\ll \hat\lambda$ (recall Lemma 3.1), it follows that

\begin{align*}1=Q_{\beta,\hat\pi}(C_n|x)+Q_{\beta,\hat\pi}(\mathcal{C}^{\beta,\hat\pi}-C_n|x)=Q_{\beta,\hat\pi}(C_n|x),\end{align*}

giving the result.

We conclude this section by proposing a sufficient condition for Assumption B. It imposes a drift towards a closed set with $\hat\lambda$ -null intersection with G by using some standard Foster–Lyapunov criteria. Such conditions are well known in the field of Markov chains. They provide easily verifiable conditions for testing stability (existence of invariant measure, recurrence, ergodicity, etc.).

Under these conditions, Assumption B is satisfied.

Proof. Let $0<\beta<1$ and $\hat{\pi}\in\hat{\boldsymbol{\Pi}}_{s}$ . The inequality (3.3) implies that

\begin{align*}\int_{\hat{\textbf{X}}-\mathcal{C}^{\beta,\hat{\pi}}} W^{\beta,\hat{\pi}}(y) Q_{\beta,\hat{\pi}}(dy|x) \leq W^{\beta,\hat{\pi}}(x)+\gamma^{\beta,\hat{\pi}}\quad\hbox{for $\hat{\lambda}$-almost all $x\in\hat{\textbf{X}}$}.\end{align*}

But since $W^{\beta,\hat\pi}(y)$ is infinite when $y\notin\mathcal{C}^{\beta,\hat\pi}$ and $W^{\beta,\hat\pi}(x)$ is finite on $\mathcal{C}^{\beta,\hat\pi}$ , this implies that $Q_{\beta,\hat{\pi}}(\hat{\textbf{X}}-\mathcal{C}^{\beta,\hat{\pi}}|x)=0$ and so $Q_{\beta,\hat{\pi}}(\mathcal{C}^{\beta,\hat{\pi}}|x)=1$ for $\hat{\lambda}$ -almost all $x\in\mathcal{C}^{\beta,\hat{\pi}}$ . This establishes the first part of Assumption B. Now, following the proof of Lemma 11.3.6 in [Reference Meyn and Tweedie22] and combining (2.1) and (3.5), we get $\mathbb{E}^{\beta,\hat{\pi}}_{x}[\tau_{\mathcal{C}^{\beta,\hat{\pi}}}] \leq V^{\beta,\hat{\pi}}(x) $ for $\hat{\lambda}$ -almost all $x\in\hat{\textbf{X}}-\mathcal{C}^{\beta,\hat{\pi}}$ . The stated result follows from (3.4).

We note that the sufficient conditions given in Proposition 3.2 above are, in general, more easily verifiable than Assumption B. Indeed, Proposition 3.2 involves just the one-step transitions of the system (see (3.3) and (3.5)). On the other hand, checking Assumption B directly requires one to compute the expected hitting time of the set $\mathcal{C}^{\beta,\hat\pi}$ , which in principle requires knowledge of the distribution of the Markov chain over the whole time horizon, and then check that these expected hitting times are bounded uniformly in the initial state of the system.

It is worth mentioning that the condition (3.3) yields, loosely, the partition between recurrent and transient states of the Markov chain, while (3.5) is closely related to the computation of the expected absorption time by the set of recurrent states of the Markov chain.

Next, in Example 3.1, we illustrate the verification of the conditions in Proposition 3.2; see also Example 4.1 below.

Example 3.1. (Taken from [Reference Dufour and Prieto-Rumeau15, Example 4.6].) Consider the state space $\textbf{X}=\{0,1,2\}$ with no actions at states 0 and 2, and $\textbf{A}(1)=\{a_1,a_2\}$ . The transition probabilities from state 0 are $q(0,0)=q(0,2)=1/2$ , and from state 2 they are $q(2,2)=1$ (we do not make any action explicit in this notation, since there are no actions at 0 and 2). The transitions from state 1 are $q(1,a_1,0)=2/3$ and $q(1,a_1,1)=1/3$ under $a_1$ , and $q(1,a_2,0)=q(1,a_2,1)=$ $q(1,a_2,2)=1/3$ under $a_2$ . We let $G=\{0\}$ . We can identify $\hat{\boldsymbol{\Pi}}_s$ with the interval [0, 1], where $a\in[0, 1]$ is the probability of selecting $a_1$ and $1-a$ is the probability of choosing action $a_2$ . Let 1 be the initial state of the system.

In order to verify Assumption B, we will use the sufficient conditions in Proposition 3.2. After adding the cemetery state $\Delta$ , the transition matrix for the stationary policy $a\in[0, 1]$ in the model $\mathcal{M}_{\beta}$ is

\begin{align*}\begin{pmatrix} \beta/2 & \quad 0 & \quad \beta/2 & \quad 1-\beta \\(1+a)/3 & \quad 1/3 & \quad (1-a)/3 & \quad 0 \\0 & \quad 0 & \quad 1 & \quad 0 \\0 & \quad 0 & \quad 0 & \quad 1\end{pmatrix},\end{align*}

where the states are ordered $0,1,2,\Delta$ .

Letting $W^{\beta,a}=(+\infty,+\infty,1,1)$ and $\gamma^{\beta,a}=1$ , we observe that (3.3) in Proposition 3.2 holds with, therefore, $\mathcal{C}^{\beta,a}=\{2,\Delta\}$ (roughly, W equals 1 on the recurrent states of the Markov chain, and it is infinite on the transient states). Regarding the inequalities (3.5), we obtain their minimal nonnegative solution, which is

\begin{align*}V^{\beta,a}(0)=\frac{2}{2-\beta},\qquad V^{\beta,a}(1)=\frac{(1+a)}{2-\beta}+\frac{3}{2},\qquad V^{\beta,a}(2)=V^{\beta,a}(\Delta)=0.\end{align*}

This minimal nonnegative solution is precisely the expected time the Markov chain spends in the transient states.

Solving the total-expected-reward MDP model $\mathcal{M}_{\beta}$ yields the optimal policy $\hat\pi^*_{\beta}=1$ with $V^*_{\beta}(1)=\frac{2}{2-\beta}+\frac{3}{2}$ for every $0<\beta<1$ .

4. Main results

In this section we prove our results on the solution of the control model $\mathcal{M}$ .

Proposition 4.1. Under Assumptions A and B, for each $0<\beta<1$ the mapping

\begin{align*}P\mapsto \int_{\boldsymbol{\Omega}} (1-\beta^{N_G})\textrm{d} P\end{align*}

is lower semicontinuous on $\boldsymbol{\mathcal{S}}_{\nu}$ and continuous on $\boldsymbol{\mathcal{S}}_{\nu}^{s}$ .

Proof. First we prove the following preliminary fact:

(4.1) \begin{eqnarray}\lim_{n\rightarrow\infty}\sup_{\hat\pi\in\hat{\boldsymbol{\Pi}}_s} \mathbb{E}^{\beta,\hat{\pi}}_{\nu}[\hat{N}_G([n,\infty))]=0\end{eqnarray}

for any $0<\beta<1$ . Observe that for arbitrary $\hat{\pi}\in\hat{\boldsymbol{\Pi}}_{s}$ and $m\geq n\ge0$ ,

\begin{eqnarray*}\mathbb{P}^{\beta,\hat{\pi}}_{\nu}\{\tau_{\mathcal{C}^{\beta,\hat{\pi}}}=n,\hat{X}_{m}\in G\}&\,\leq\,&\mathbb{P}^{\beta,\hat\pi}_{\nu}\{\hat{X}_{n}\in\mathcal{C}^{\beta,\hat{\pi}},\hat{X}_{m}\in G \}\\&=& \,\mathbb{E}^{\beta,\hat\pi}_{\nu}[ Q^{m-n}_{\beta,\hat{\pi}}(G|\hat{X}_{n}) \textbf{I}_{\mathcal{C}^{\beta,\hat{\pi}}}(\hat{X}_{n})].\end{eqnarray*}

Recalling that $\hat{\lambda}(G\cap\mathcal{C}^{\beta,\hat{\pi}})=0$ , Lemmas 3.1 and 3.2 imply that the above expectation vanishes, and so $\mathbb{P}^{\beta,\hat{\pi}}_{\nu}\{\tau_{\mathcal{C}^{\beta,\hat{\pi}}}=n,\hat{X}_{m}\in G\}=0$ . Therefore, we have

\begin{align*}\hat{N}_G([n,\infty))= \textbf{I}_{\{\tau_{\mathcal{C}^{\beta,\hat{\pi}}}>n\}} \sum_{m=n}^{\tau_{\mathcal{C}^{\beta,\hat{\pi}}}-1}\textbf{I}_{G}(\hat{X}_{m}) \le (\tau_{\mathcal{C}^{\beta,\hat{\pi}}}-n)^+ \quad \mathbb{P}^{\beta,\hat{\pi}}_{\nu}\text{-almost surely,}\end{align*}

and so

\begin{eqnarray*} \mathbb{E}^{\beta,\hat{\pi}}_{\nu}[\hat{N}_G([n,\infty))] &\,\le\, & \sum_{m>n}^\infty(m-n) \mathbb{P}^{\beta,\hat{\pi}}_{\nu} \{\tau_{\mathcal{C}^{\beta,\hat{\pi}}}=m\} \\&=& \, \sum_{m>n}^\infty(m-n) \mathbb{E}^{\beta,\hat{\pi}}_{\nu} \big[ \textbf{I}_{\hat{\textbf{X}}-\mathcal{C}^{\beta,\hat{\pi}}}(\hat{X}_{n}) \;\mathbb{P}^{\beta,\hat{\pi}}_{\hat{X}_{n}} \{\tau_{\mathcal{C}^{\beta,\hat{\pi}}}=m-n\} \big]\end{eqnarray*}

by the Markov property. Then it follows that

\begin{eqnarray*} \mathbb{E}^{\beta,\hat{\pi}}_{\nu}[\hat{N}_G([n,\infty))] &\,\le\, & \mathbb{E}^{\beta,\hat{\pi}}_{\nu} \big[ \textbf{I}_{\hat{\textbf{X}}-\mathcal{C}^{\beta,\hat{\pi}}}(\hat{X}_{n}) \cdot \sum_{m>n}^\infty(m-n) \mathbb{P}^{\beta,\hat{\pi}}_{\hat{X}_{n}} \{\tau_{\mathcal{C}^{\beta,\hat{\pi}}}=m-n\} \big] \\ &=& \, \mathbb{E}^{\beta,\hat{\pi}}_{\nu} \big[ \textbf{I}_{\hat{\textbf{X}}-\mathcal{C}^{\beta,\hat{\pi}}}(\hat{X}_{n}) \cdot\mathbb{E}^{\beta,\hat{\pi}}_{\hat{X}_{n}} [\tau_{\mathcal{C}^{\beta,\hat{\pi}}} ] \big] \\ &\le&\, \mathbb{E}^{\beta,\hat{\pi}}_{\nu} \big[ \textbf{I}_{\hat{\textbf{X}}-\mathcal{C}^{\beta,\hat{\pi}}}(\hat{X}_{n}) \cdot K_{\beta}\big],\end{eqnarray*}

where the last inequality is obtained because $\mathbb{E}^{\beta,\hat{\pi}}_{x} [\tau_{\mathcal{C}^{\beta,\hat{\pi}}} ]\le K_{\beta}$ for $\hat\lambda$ -almost every $x\in\hat{\textbf{X}}-\mathcal{C}^{\beta,\hat\pi}$ —recall (3.2)—and because the distribution of $\hat X_n$ is absolutely continuous with respect to $\hat\lambda$ —see Lemma 3.1. From the Markov inequality we derive that

\begin{align*} \mathbb{E}^{\beta,\hat{\pi}}_{\nu}[\hat{N}_G([n,\infty))] \leq K_{\beta} \cdot \mathbb{P}^{\beta,\hat{\pi}}_{\nu} \{\tau_{\mathcal{C}^{\beta,\hat{\pi}}}>n\} \leq K_{\beta}^2 /n;\end{align*}

note that this bound does not depend on $\hat\pi\in\hat{\boldsymbol{\Pi}}_s$ . This shows (4.1).

We now proceed with the proof. For each fixed $n\ge0$ , the mapping on $(\textbf{X}\times\textbf{A})^{n-1}\times\textbf{X}$ given by

\begin{align*}(x_0,a_0,\ldots,x_n)\mapsto\prod_{0\le j\le n}\beta^{\textbf{I}_{G}(x_j)}=\beta^{N_G([0,n])}\end{align*}

is in $\boldsymbol{\mathcal{U}}_n$ . Therefore, by the definition of the $ws^\infty$ -topology, the function

(4.2) \begin{equation}P\mapsto \int_{\boldsymbol{\Omega}} (1-\beta^{N_G([0,n])})\textrm{d} P\end{equation}

is continuous on $\boldsymbol{\mathcal{S}}_{\nu}$ , and so the function $P\mapsto\int_{\boldsymbol{\Omega}}(1-\beta^{N_G})\textrm{d} P$ is lower semicontinuous on $\boldsymbol{\mathcal{S}}_{\nu}$ , since it is the increasing limit as $n\rightarrow\infty$ of continuous functions.

Now we prove upper semicontinuity on $\boldsymbol{\mathcal{S}}_{\nu}^{s}$ . Fix arbitrary $0\le n\le m$ and $P\in \boldsymbol{\mathcal{S}}_{\nu}^{s}$ such that $P=\mathbb{P}^{\pi}_{\nu}$ for some $\pi\in\boldsymbol{\Pi}_{s}$ . Using Proposition 3.1,

\begin{eqnarray*}\int_{\boldsymbol{\Omega}} (1-\beta^{N_G([0,m])}) \textrm{d}\mathbb{P}^{\pi}_{\nu}\, &=& \, (1-\beta)\mathbb{E}^{\beta,\pi^{\Delta}}_{\nu}[\hat{N}_G([0,m])] \\&\le&\, (1-\beta) \mathbb{E}^{\beta,\pi^{\Delta}}_{\nu}[\hat{N}_G([0,n])] \\&&+\, (1-\beta)\sup_{\hat{\pi}\in\hat{\boldsymbol{\Pi}}_{s}} \mathbb{E}^{\beta,\hat\pi}_{\nu}[\hat{N}_G([n+1,\infty))]\\&=& \, \int_{\boldsymbol{\Omega}} (1-\beta^{N_G([0,n])})\textrm{d} \mathbb{P}^{\pi}_{\nu}+\epsilon_n,\end{eqnarray*}

where we have defined

\begin{align*}\epsilon_n= (1-\beta)\sup_{\hat{\pi}\in\hat{\boldsymbol{\Pi}}_{s}} \mathbb{E}^{\beta,\hat{\pi}}_{\nu}[\hat{N}_G([n+1,\infty))],\end{align*}

with $\epsilon_n\rightarrow0$ as a consequence of (4.1). By the result in [Reference Schäl27, Proposition 10.1], it follows that the limit as $n\rightarrow\infty$ of the function (4.2) is an upper semicontinous function on $\boldsymbol{\mathcal{S}}_{\nu}^{s}$ . This completes the proof of the continuity statement on $\boldsymbol{\mathcal{S}}_{\nu}^{s}$ .

Proposition 4.2. Suppose that Assumptions A and B are satisfied and fix $0<\beta<1$ . There exists a stationary policy $\hat{\pi}^*$ in $\hat{\boldsymbol{\Pi}}_{s}$ that is optimal for the control model $\mathcal{M}_{\beta}$ , i.e.,

\begin{align*}V^*_{\beta} (\nu)=\sup_{\hat{\pi}\in\hat{\boldsymbol{\Pi}}} \mathbb{E}^{\beta,\hat{\pi}}_{\nu} [\hat{N}_G]= \mathbb{E}^{\beta,\hat{\pi}^*}_{\nu}[\hat{N}_G].\end{align*}

Proof. Using the result in [Reference Schäl30, p. 368], we have

\begin{align*}\sup_{\hat{\pi}\in\hat{\boldsymbol{\Pi}}} \mathbb{E}^{\beta,\hat{\pi}}_{\nu}[\hat{N}_G]=\sup_{\hat{\pi}\in\hat{\boldsymbol{\Pi}}_{s}} \mathbb{E}^{\beta,\hat{\pi}}_{\nu}[\hat{N}_G].\end{align*}

Consequently, from Proposition 3.1 we obtain

\begin{align*} \sup_{\pi\in\boldsymbol{\Pi}} \mathbb{E}^{\pi}_{\nu}[1-\beta^{N_G}]=\sup_{\pi\in\boldsymbol{\Pi}_{s}} \mathbb{E}^{\pi}_{\nu}[1-\beta^{N_G}]=\sup_{P\in\boldsymbol{\mathcal{S}}_{\nu}^{s}} \int (1-\beta^{N_G})\textrm{d} P.\end{align*}

But the latter supremum is attained at some $P^*\in\boldsymbol{\mathcal{S}}^s_{\nu}$ , because it is the maximum of a continuous function (Proposition 4.1) on a compact set (Theorem 2.1). If $P^*=\mathbb{P}^{\pi_*}_{\nu}$ for some $\pi_*\in\boldsymbol{\Pi}_s$ , then we have that $\pi_*^\Delta\in\hat{\boldsymbol{\Pi}}_s$ is an optimal stationary Markov policy for the control problem  $\mathcal{M}_{\beta}$ .

We are now in position to prove the main result of this paper, on maximizing the probability of visiting the set G infinitely often.

Proof. (i) Let $\{\beta_k\}_{k\in\mathbb{N}}$ be a sequence in (0, 1) with $\beta_k\uparrow1$ , and let the policy $\hat{\pi}_{k}\in\hat{\boldsymbol{\Pi}}_{s}$ be optimal for $\mathcal{M}_{\beta_k}$ . Such a policy exists by Proposition 4.2. Consider now the sequence $\{\pi_{k}\}_{k\in\mathbb{N}}$ in $\boldsymbol{\Pi}_{s}$ where $\pi_{k}=\hat{\pi}^{|\textbf{X}}_{k}$ for $k\in\mathbb{N}$ . The set $\boldsymbol{\Pi}_s$ being compact (Lemma 2.2), we have that $\{\pi_{k}\}_{k\in\mathbb{N}}$ has a convergent subsequence, and we can assume without loss of generality that the whole sequence $\{\pi_{k}\}_{k\in\mathbb{N}}$ converges to some $\pi\in\boldsymbol{\Pi}_s$ . In particular, by Lemma 2.3, we also have $\mathbb{P}_{\nu}^{\pi_{k}}\Rightarrow \mathbb{P}_{\nu}^{\pi}$ .

By Proposition 3.1 it follows that for arbitrary $\gamma\in\boldsymbol{\Pi}$ ,

\begin{eqnarray*}\mathbb{E}_{\nu}^{\pi_k}[1-\beta_k^{N_G}] &=& \, (1-\beta_k) \mathbb{E}^{\beta_k,\hat{\pi}_{k}}_{\nu} [\hat{N}_G]\ =\ (1-\beta_k)V^*_{\beta_k}(\nu) \\ &\geq& (1-\beta_k) \mathbb{E}^{\beta_k,\gamma^{\Delta}}_{\nu} [\hat{N}_G]\ =\ \mathbb{E}_{\nu}^{\gamma}[1-\beta_k^{N_G}].\end{eqnarray*}

Now, by the first statement in Remark 3.2(b), observe that

\begin{align*}\mathbb{E}^{\gamma}_{\nu} [1-\beta_k^{N_G}] \ge\mathbb{P}^{\gamma}_{\nu}\{N_{G}=\infty\},\end{align*}

and so for any $j\leq k$ in $\mathbb{N}$ we obtain the inequalities

\begin{align*}\mathbb{E}_{\nu}^{\pi_k}[1-\beta_j^{N_G}]\ge \mathbb{E}_{\nu}^{\pi_k}[1-\beta_k^{N_G}]=(1-\beta_k)V^*_{\beta_k}(\nu)\ge \mathbb{P}^{\gamma}_{\nu}\{N_{G}=\infty\}.\end{align*}

Taking the limit as $k\rightarrow\infty$ with j fixed in the previous inequalities, we get from Proposition 4.1 that

\begin{equation*}\mathbb{E}_{\nu}^{\pi}[1-\beta_j^{N_G}]\ge \limsup_{k\rightarrow\infty}(1-\beta_k)V^*_{\beta_k}(\nu)\ge \liminf_{k\rightarrow\infty} (1-\beta_k)V^*_{\beta_k}(\nu)\ge\mathbb{P}_{\nu}^{\gamma}\{N_{G}=\infty\}.\end{equation*}

In these inequalities, we take the limit as $j\rightarrow\infty$ and the supremum in $\gamma\in\boldsymbol{\Pi}$ to obtain

\begin{equation*}\mathbb{P}_{\nu}^{\pi}\{N_{G}=\infty\}\ge \limsup_{k\rightarrow\infty}(1-\beta_k)V^*_{\beta_k}(\nu)\ge \liminf_{k\rightarrow\infty} (1-\beta_k)V^*_{\beta_k}(\nu)\ge\sup_{\gamma\in\boldsymbol{\Pi}}\mathbb{P}_{\nu}^{\gamma}\{N_{G}=\infty\}.\end{equation*}

Consequently, the above inequalities are all, in fact, equalities:

(4.3) \begin{equation}\mathbb{P}_{\nu}^{\pi}\{N_{G}=\infty\}=V^*(\nu)= \lim_{k\rightarrow\infty}(1-\beta_k)V^*_{\beta_k}(\nu).\end{equation}

This establishes that the stationary Markov policy $\pi\in\boldsymbol{\Pi}_s$ is optimal for the control model $\mathcal{M}$ and that $(1-\beta_k)V^*_{\beta_k}(\nu)\rightarrow V^*(\nu)$ .

(ii) We can obtain the proof of this statement from the proof of (i), just by assuming that the whole sequence $\{\pi_k\}$ converges (and not invoking a convergent subsequence).

(iii) Using Proposition 3.1 we obtain that

\begin{align*}(1-\beta)V^*_{\beta}(\nu)=\sup_{\gamma\in\boldsymbol{\Pi}} \mathbb{E}_{\nu}^{\gamma}[1-\beta^{N_G}].\end{align*}

This implies that $(1-\beta)V^*_{\beta}(\nu)$ decreases as $\beta\uparrow1$ , and so $\lim_{\beta\uparrow1} (1-\beta)V^*_{\beta}(\nu)$ exists. Since we have convergence through some sequence of $\beta_k\uparrow1$ (recall (4.3)), the stated result follows.

Summarizing, there indeed exists a stationary Markov policy in $\boldsymbol{\Pi}_{s}$ that maximizes the probability of visiting the set G infinitely often. This policy is obtained as an accumulation/limit point as $\beta\uparrow1$ in the sense of convergence of Young measures of optimal stationary Markov policies for the total-expected-reward problem $\sup_{\hat{\pi}\in\hat{\boldsymbol{\Pi}}} \mathbb{E}_{\nu}^{\beta,\hat{\pi}}[\hat{N}_G]$ of the control model $\mathcal{M}_{\beta}$ .

Example 4.1. Consider a control model with state space $\textbf{X}=\{0,c,g,g_0,g_1,g_2,\ldots\}$ . There are two actions available at state 0, i.e. $\textbf{A}(0)=\{a_1,a_2\}$ , and there are no actions at the other states. The transition probabilities are $q(0,a_1,g_0)=1$ , $q(0,a_2,c)=q(0,a_2,g)=1/2$ . From state $g_i$ , the transition probabilities are

\begin{align*}q(g_i,g_{i+1})=\frac{1}{i+1}\quad\hbox{and}\quad q(g_i,c)=\frac{i}{i+1}\quad\hbox{for $i\ge0$},\end{align*}

while c and g are absorbing. Let $G=\{g,g_0,g_1,\ldots\}$ and let 0 be the initial state of the system. It should be clear that Assumption A holds.

Consider now the control model $\mathcal{M}_{\beta}$ . We identify $\hat{\boldsymbol{\Pi}}_s$ with the interval [0, 1], where $0\le a\le 1$ stands for the probability of choosing action $a_1$ at state 0. Let us verify the conditions in Proposition 3.2. Concerning (3.3), we put $W^{\beta,a}(c)=W^{\beta,a}(\Delta)=1$ and we let $W^{\beta,a}$ be $+\infty$ in the remaining states, thus obtaining $\mathcal{C}^{\beta,a}=\{c,\Delta\}$ , with $\gamma^{\beta,a}=1$ . This identifies the transient and recurrent states of the Markov chain. For (3.5), we let $V^{\beta,a}(c)=V^{\beta,a}(\Delta)=0$ , and we try to find the minimal solution of (3.5). These inequalities become

\begin{align*} \frac{1}{2}(1-a)V^{\beta,a}(g)+a V^{\beta,a}(g_0)\le V^{\beta,a}(0)-1,\quad \beta V^{\beta,a}(g)\le V^{\beta,a}(g)-1, \end{align*}

and

\begin{align*} \frac{\beta}{i+1}V^{\beta,a}(g_{i+1})\le V^{\beta,a}(g_i)-1\quad\hbox{for $i\ge0$}.\end{align*}

By iteration of the latter inequalities we obtain

\begin{align*}\frac{\beta^{i+1}}{(i+1)!}V^{\beta,a}(g_{i+1})\le V^{\beta,a}(g_0)-\sum_{j=0}^i \frac{\beta^j}{j!}.\end{align*}

The minimal solution is attained when $V^{\beta,a}(0)=e^\beta$ , and thus for any $i\ge0$

\begin{align*}V^{\beta,a}(g_{i})=1+\frac{\beta}{k+1}+\frac{\beta^2}{(k+2)(k+1)}+\frac{\beta^3}{(k+3)(k+2)(k+1)}+\ldots,\end{align*}

$V^{\beta,a}(g)=\frac{1}{1-\beta}$ , and

\begin{align*}V^{\beta,a}(0)=1+\frac{1-a}{2(1-\beta)}+ae^\beta.\end{align*}

We note that this minimal solution $V^{\beta,a}(x)$ is precisely the expected time the process spends in the transient states of the chain when starting from $x\in\textbf{X}$ . We also note that $V^{\beta,a}(x)$ is bounded in x (recall (3.4)): to see this, observe that $V^{\beta,a}(g_i)\le e^\beta$ for every $i\ge0$ . It should be clear that the optimal value function of the control model $\mathcal{M}_{\beta}$ for the initial state 0 is

\begin{align*}V^*_{\beta}(0)=\max\Big\{\frac{1}{2(1-\beta)},e^\beta\Big\}\end{align*}

and that the optimal policy is $\hat\pi^*_{\beta}=1$ for $0<\beta\le \hat\beta$ and $\hat\pi^*_{\beta}=0$ for $\hat\beta\le \beta<1$ , where $\hat\beta\simeq 0.76804$ is the unique solution in (0, 1) of $\frac{1}{2(1-z)}=e^{z}$ . The interesting feature of this example is that the optimal policy $\pi^*_{\beta}$ of $\mathcal{M}_{\beta}$ varies with $\beta$ (cf. Example 3.1).

From Theorem 4.1(ii)–(iii) we obtain that

\[0=\pi^*=\lim_{\beta\uparrow1} \hat\pi^{*|\textbf{X}}_{\beta}\]

is an optimal policy for $\mathcal{M}$ and that $V^*(1)=\lim_{\beta\uparrow1} (1-\beta)V^*_{\beta}(1)=1/2$ .