Nomenclature

$\alpha $ , $\beta $

attack angle and slide angle

${\omega _z}$ , ${\omega _y}$

angle velocities

${J_z}$ , ${J_y}$

rotational inertia

${\delta _z}$ , ${\delta _y}$

output signals of the elevator and rudder

m

mass of the missile

V

velocity of the missile

S

reference area of the missile

L

reference length of the missile

$C_{\left( \cdot \right)}^{\left( \cdot \right)}$

coefficients of the aerodynamic forces

$m_{\left( \cdot \right)}^{\left( \cdot \right)}$

coefficients of the aerodynamic moments

${d_i}\!\left( {i = {\omega _z},{\omega _y}} \right)$

unknown disturbances

${\chi _i}\!\left( {i = z,y} \right)$

uncertainties caused by the unmodeled dynamics

$\eta \!\left( t \right)$

unmodeled dynamics

${y_d}\!\left( t \right)$

desired tracking signal

${x_{2c}}\!\left( t \right)$

inner loop virtual signal

${e_1}\!\left( t \right)$ , ${e_2}\!\left( t \right)$

tracking errors

${k_0}$ , ${k_1}$ , ${k_2}$

positive control gains

$r\!\left( t \right)$

dynamic auxiliary signal

${\rm{\Delta }}f$

system unknown nonlinear term

${W_a}$ , ${W_c}$

weights of actor NN and critic NN

${{\rm{\Phi }}_a}\!\left( {{Z_a}} \right)$ , ${{\rm{\Phi }}_c}\!\left( {{Z_c}} \right)$

activation functions of actor NN and critic NN

${\varepsilon _{{W_a}}}$ , ${\varepsilon _{{W_c}}}$

estimation errors of actor NN and critic NN

${\varepsilon _v}$

unknown upper bound of the total disturbance $D\!\left( t \right)$

$u\!\left( t \right)$

designed controller

$J\!\left( t \right)$

integral penalty function

${e_c}\!\left( t \right)$

error variable of critic NN

${E_c}\!\left( t \right)$

error objective function of critic NN

$\hat \cdot $

estimation value of $ \cdot $

$\tilde \cdot $

estimation error of $ \cdot $ and $\tilde \cdot = \hat \cdot - \cdot $

2.0 Problem formulation and preliminaries

2.1 Problem statement

Ignoring the roll channel, the attitude dynamic model of a direct force and aerodynamic force compound missile can be modeled as

(1) \begin{align}\begin{array}{l} \dot \alpha = {\omega _z} - \dfrac{{QS\!\left( {C_y^\alpha \alpha + C_y^{{\delta _z}}{\delta _z}} \right)}}{{mV}} - \dfrac{{{F_y}}}{{mV}}\\[5pt] \dot \beta = {\omega _y} + \dfrac{{QS\!\left( {C_z^\beta \beta + C_z^{{\delta _y}}{\delta _y}} \right)}}{{mV}} + \dfrac{{{F_z}}}{{mV}}\\[5pt] {{\dot \omega }_z} = \dfrac{{QSL}}{{{J_z}}}\!\left( {m_z^\alpha \alpha + m_z^{{\delta _z}}{\delta _z} + m_z^{{{\bar \omega }_z}}{{\bar \omega }_z}} \right) + \dfrac{{{l_z}}}{{{J_z}}}{F_y} + {d_{{\omega _z}}} + {\chi _z}\!\left( {{\omega _z},\eta } \right)\\[5pt] {{\dot \omega }_y} = \dfrac{{QSL}}{{{J_z}}}\!\left( {m_y^\beta \beta + m_y^{{\delta _y}}{\delta _y} + m_y^{{{\bar \omega }_y}}{{\bar \omega }_y}} \right) + \dfrac{{{l_y}}}{{{J_y}}}{F_z} + {d_{{\omega _y}}} + {\chi _y}\!\left( {{\omega _y},\eta } \right)\end{array} \end{align}

where $\alpha $ is the angle-of-attack, $\beta $ is the slide angle, ${\omega _z}$ , ${\omega _y}$ are the angle velocities. ${J_z}$ , ${J_y}$ denote the rotational inertia. $V$ and $m$ are the velocity and the mass of the missile, respectively. $S$ and $L$ represent the reference area and length. ${\delta _z}$ , ${\delta _y}$ are the output signals of the elevator and rudder. $C_{\left( \cdot \right)}^{\left( \cdot \right)}$ denote the coefficients of the aerodynamic forces, while $m_{\!\left( \cdot \right)}^{\!\left( \cdot \right)}$ represent the coefficients of the aerodynamic moments. ${d_i}\!\left( {i = {\omega _z},{\omega _y}} \right)$ are the unknown disturbances. $\eta $ is the unmodeled dynamics, and ${\chi _i}\!\left( {i = z,y} \right)$ represents the uncertainties caused by the unmodeled dynamics.

Then, by defining

(2) \begin{align}\begin{array}{l} {x_1}\!\left( t \right) = {\left[\! {\begin{array}{*{20}{c}} \alpha &\beta \end{array}} \right]^T},{x_2}\!\left( t \right) = {\left[\! {\begin{array}{*{20}{c}} {{\omega _z}}&{{\omega _y}} \end{array}} \right]^T},\\[9pt] {d_1}\!\left( t \right) = \left[ {\begin{array}{*{20}{c}} { - \dfrac{{QS\!\left( {C_y^\alpha \alpha + C_y^{{\delta _z}}{\delta _z}} \right)}}{{mV}} - \dfrac{{{F_y}}}{{mV}}}\\[9pt] {\dfrac{{QS\!\left( {C_z^\beta \beta + C_z^{{\delta _y}}{\delta _y}} \right)}}{{mV}} + \dfrac{{{F_z}}}{{mV}}} \end{array}} \right],{d_2}\!\left( t \right) = \left[ {\begin{array}{*{20}{c}} {\dfrac{{QSL}}{{{J_z}}}\!\left( {m_z^\alpha \alpha {\rm{ + }}m_z^{{{\bar \omega }_z}}{{\bar \omega }_z}} \right) + {d_{{\omega _z}}}}\\[9pt] {\dfrac{{QSL}}{{{J_z}}}\!\left( {m_y^\beta \beta {\rm{ + }}m_y^{{{\bar \omega }_y}}{{\bar \omega }_y}} \right) + {d_{{\omega _y}}}} \end{array}} \right]\\[9pt] B = \left[ {\begin{array}{*{20}{c}} {\dfrac{{QSL}}{{{J_z}}}m_z^{{\delta _z}}}&0&{\dfrac{{{l_z}}}{{{J_z}}}}&0\\[9pt] 0&{\dfrac{{QSL}}{{{J_z}}}m_y^{{\delta _y}}}&0&{\dfrac{{{l_y}}}{{{J_y}}}} \end{array}} \right],u\!\left( t \right) = {\left[ {\begin{array}{*{20}{c}} {{\delta _z}}&{{\delta _y}}&{{F_y}}&{{F_z}} \end{array}} \right]^T}\end{array} \end{align}

the equivalent model of Equation (1) can be given as

(3) \begin{align}\begin{array}{l}{{\dot x}_1}\!\left( t \right) = {x_2}\!\left( t \right) + {d_1}\!\left( t \right)\\[5pt] {{\dot x}_2}\!\left( t \right) = Bu\!\left( t \right) + {d_2}\!\left( t \right) + \chi \!\left( {{x_2}\!\left( t \right),\eta \!\left( t \right)} \right)\end{array}\end{align}

Thus, the design objective is to develop a reinforcement learning-based STDO control scheme to maintain the desired trajectory tracking for the straight air compound missile system given in Equation (3) subjected to aerodynamic uncertainties and unmodeled dynamics.

2.2 Assumptions and lemmas

The following assumptions and lemmas are necessary.

Assumption 1. The disturbance moments caused by structural uncertainty are bounded, that is, there exists the constants ${\bar d_1}$ , ${\bar d_2}$ such that $\|{d_1}\!\left( t \right)\| \le {\bar d_1}$ , $\|{d_2}\!\left( t \right)\| \le {\bar d_2}$ .

Assumption 2. The desired tracking signal of the system ${y_d}\!\left( t \right)$ is smooth and twice differentiable.

Lemma 1. For any constant $\varepsilon \gt 0$ and vector $\xi \in {R^n}$ , we have

(4) \begin{align}\!\left\| \xi \right\| < \frac{{{\xi ^T}\xi }}{{\sqrt {{\xi ^T}\xi + {\varepsilon ^2}} }} + \varepsilon \end{align}

Lemma 2. [Reference Polycarpou and Ioannou27] Given any constant $\varepsilon \gt 0$ and any variable $z \in R$ , the following inequality holds

(5) \begin{align}0 \le \!\left| z \right| - z\tanh \!\left( {\frac{z}{\varepsilon }} \right) \le \kappa \varepsilon \end{align}

where $\kappa $ is a constant satisfying $\kappa = {{\rm{e}}^{ - \!\left( {\kappa + 1} \right)}}$ , i.e. $\kappa = 0.2785$ .

3.0 Stdo-based adaptive reinforcement learning control

In this section, as shown in Fig. 1, a STDO-based adaptive reinforcement learning control method is proposed.

Figure 1. The structure of the proposed reinforcement learning-based STDO control algorithm.

Defining the control expected output signal as ${y_d}\!\left( t \right)$ and the inner loop virtual signal as ${x_{2c}}\!\left( t \right)$ , then the tracking errors of ${x_1}\!\left( t \right)$ and ${x_2}\!\left( t \right)$ can be expressed as

(6) \begin{align}\begin{array}{l}{e_1}\!\left( t \right) = {x_1}\!\left( t \right) - {y_d}\!\left( t \right)\\[5pt] {e_2}\!\left( t \right) = {x_2}\!\left( t \right) - {x_{2c}}\!\left( t \right)\end{array}\end{align}

Combining with Equation (3), one has

(7) \begin{align}\begin{array}{l}{{\dot e}_1}\!\left( t \right) = {x_{2c}}\!\left( t \right) + {e_2}\!\left( t \right) + {d_1}\!\left( t \right) - {{\dot y}_d}\!\left( t \right)\\[5pt] {{\dot e}_2}\!\left( t \right) = Bu\!\left( t \right) + {d_2}\!\left( t \right) + \chi \!\left( {{x_2}\!\left( t \right),\eta \!\left( t \right)} \right) - {{\dot x}_{2c}}\!\left( t \right)\end{array}\end{align}

Then, we can design the inner loop virtual signal as

(8) \begin{align} {x_{2c}}\!\left( t \right) = - {k_0}\int_0^t {{e_1}\!\left( \tau \right)d\tau } - {k_1}{e_1}\!\left( t \right) - \hat{d}_{1}\!\left( t \right) + \dot{y}_{d}\!\left( t \right) \end{align}

where ${\hat{d}_1}\!\left( t \right)$ is the adaptive estimate of ${d_1}\!\left( t \right)$ , ${k_0}$ and ${k_1}$ are the control gains.

In order to compensate and suppress the influence of the unknown mismatched disturbance ${d_1}\!\left( t \right)$ , a second-order STDO is designed as follows

(9) \begin{align} \begin{array}{l}\dot{\hat d}_1\!\left( t \right) = - {K_{{d_1}}}\!\left( {{{\hat d}_1}\!\left( t \right) - {P_1}} \right)\\[5pt] {P_1} = - {K_{{P_1}}}\dfrac{{{{\hat x}_1} - {x_1}}}{{{{\!\left\| {{{\hat x}_1} - {x_1}} \right\|}^{\frac{1}{2}}}}} + {P_2}\\[5pt] {{\dot P}_2} = - {K_{{P_2}}}\dfrac{{{{\hat x}_1} - {x_1}}}{{\!\left\| {{{\hat x}_1} - {x_1}} \right\|}}\\[5pt] \dot {\hat x}_1 \!\left( t \right) = {x_2}\!\left( t \right) + {{\hat d}_1}\!\left( t \right)\end{array} \end{align}

The dynamic signal $r\!\left( t \right)$ is introduced, which is defined by

(10) \begin{align}\dot r\!\left( t \right) = - {\gamma _0}r\!\left( t \right) + \rho \!\left( {{x_1}\!\left( t \right),{x_2}\!\left( t \right)} \right),r\!\left( 0 \right) = {r_0}\end{align}

where ${\gamma _0} \in \left( {0,{\rm{\;\;}}{\gamma _1}} \right)$ .

The coupling uncertainty is assumed to satisfy the following inequality

(11) \begin{align}e_2^T\chi \!\left( {{x_2}\!\left( t \right),\eta \!\left( t \right)} \right) \le \!\left\| {e_2^T\!\left( t \right)} \right\|\!\left( {{\varphi _1}\!\left( {{x_2}\!\left( t \right)} \right) + {\varphi _2}\!\left( {\eta \!\left( t \right)} \right)} \right)\end{align}

According to Lemma 1 and Young’s inequality, Equation (11) can be rewritten as

(12) \begin{align}\begin{array}{l}\!\left\| {e_2^T\!\left( t \right)} \right\|{\varphi _1}\!\left( {{x_2}\!\left( t \right)} \right) \le e_2^T\!\left( t \right){{\bar \varphi }_1}\!\left( {{e_2}\!\left( t \right),{x_2}\!\left( t \right)} \right) + {\varepsilon _1}\\[5pt] \!\left\| {e_2^T\!\left( t \right)} \right\|{\varphi _2}\!\left( {\eta \!\left( t \right)} \right) \le e_2^T\!\left( t \right){{\bar \varphi }_2}\!\left( {{e_2}\!\left( t \right),r\!\left( t \right)} \right) + {\varepsilon _2} + \dfrac{1}{4}e_2^T\!\left( t \right){e_2}\!\left( t \right) + {\varepsilon _3}\end{array}\end{align}

where ${\varepsilon _1},{\rm{\;\;}}{\varepsilon _2}\gt 0$ are arbitrary constants,

(13) \begin{align}\begin{array}{l}{{\bar \varphi }_1}\!\left( {{e_2}\!\left( t \right),{x_2}\!\left( t \right)} \right) = \dfrac{{{\varphi _1}\!\left( {{x_2}\!\left( t \right)} \right)e_2^T\!\left( t \right){\varphi _1}\!\left( {{x_2}\!\left( t \right)} \right)}}{{\sqrt {{{\left[ {e_2^T\!\left( t \right){\varphi _1}\!\left( {{x_2}\!\left( t \right)} \right)} \right]}^2} + \varepsilon _1^2} }}\\[5pt] {{\bar \varphi }_2}\!\left( {{e_2}\!\left( t \right),r\!\left( t \right)} \right) = \dfrac{{{\varphi _2} \circ \alpha _1^{ - 1}\!\left( {2r\!\left( t \right)} \right)e_2^T\!\left( t \right){\varphi _2} \circ \alpha _1^{ - 1}\!\left( {2r\!\left( t \right)} \right)}}{{\sqrt {{{\left[ {e_2^T\!\left( t \right){\varphi _2} \circ \alpha _1^{ - 1}\!\left( {2r\!\left( t \right)} \right)} \right]}^2} + \varepsilon _2^2} }}\\[5pt] {\varepsilon _3} = {\left[ {{\varphi _2} \circ \alpha _1^{ - 1}\!\left( {2{\varepsilon _r}} \right)} \right]^2}\end{array}\end{align}

Next, we define

(14) \begin{align}\Delta f = {\bar \varphi _1}\!\left( {{e_2}\!\left( t \right),{x_2}\!\left( t \right)} \right) + {\bar \varphi _2}\!\left( {{e_2}\!\left( t \right),r\!\left( t \right)} \right)\end{align}

Since ${\bar \varphi _1}\!\left( {{e_2}\!\left( t \right),{x_2}\!\left( t \right)} \right)$ and ${\bar \varphi _2}\!\left( {{e_2}\!\left( t \right),r\!\left( t \right)} \right)$ are the functions that change irregularly in the control dynamic process, the actor NNs are introduced to approximate the unknown nonlinear term ${\rm{\Delta }}f$ , the actor NN structures of the optimal control ${\rm{\Delta }}f$ and the actual control ${\rm{\Delta }}\hat{f}$ are designed as follows

(15) \begin{align}\begin{array}{l}\Delta f = W_a^T{\Phi _a}\!\left( {{Z_a}} \right) + {\varepsilon _{{W_a}}}\\[5pt] \Delta {{\hat f}_{}} = \hat W_a^T{\Phi _a}\!\left( {{Z_a}} \right)\end{array}\end{align}

where ${W_a},{\hat W_a} \in {{\rm{R}}^{{p_1} \times n}},{{\rm{\Phi }}_a}\!\left( {{Z_a}} \right) \in {{\rm{R}}^{{p_1} \times 1}},{{\rm{\Phi }}_a}\!\left( {{Z_a}} \right) = {e^{ - \frac{{{{\!\left( {{Z_a} - \mu } \right)}^2}}}{{2{\sigma ^2}}}}},{\rm{\;\;}}{Z_a} = {\left[ {{e_2}\!\left( t \right),{x_2}\!\left( t \right),r\!\left( t \right)} \right]^T}$ , and there exists an upper bound of the estimation error such that $\|{\varepsilon _{{W_a}}}\| \le {\bar \varepsilon _{{W_a}}}$ . Thus, we can obtain that

(16) \begin{align} e_2^T\!\left( t \right)\chi \!\left( {{x_2}\!\left( t \right),\eta \!\left( t \right)} \right) \le e_2^TW_a^T{\Phi _a}\!\left( {{Z_a}} \right) + e_2^T{\varepsilon _{{W_a}}} + \frac{1}{4}e_2^T\!\left( t \right){e_2}\!\left( t \right) + \sum\limits_{i = 1}^3 {{\varepsilon _i}} \end{align}

Then, the matched disturbance ${d_2}\!\left( t \right)$ and actor NN estimation error ${\varepsilon _{{W_a}}}$ of the controlled system need to be considered and compensated. Firstly, the total disturbance $D\!\left( t \right)$ can be constructed in the following form:

(17) \begin{align}D\!\left( t \right) = {d_2}\!\left( t \right) + {\varepsilon _{{W_a}}}\end{align}

Thanks to Assumption 1, the following inequality is satisfied

(18) \begin{align}\!\left| {D\!\left( t \right)} \right| = \left| {{d_2}\!\left( t \right) + {\varepsilon _{{W_a}}}} \right| \le {\varepsilon _v}\end{align}

where ${\varepsilon _v}$ is an unknow positive constant. According to Lemma 2, we can easily obtain

(19) \begin{align}e_2^T\!\left( t \right)D\!\left( t \right) \le \!\left| {{e_2}\!\left( t \right)} \right|{\varepsilon _v} \le {\varepsilon _v}e_2^T\!\left( t \right)\tanh \!\left( {\frac{{{e_2}\!\left( t \right)}}{\alpha }} \right) + \kappa \alpha {\varepsilon _v}\end{align}

Based on the above analysis, the outer loop controller can be designed as

(20) \begin{align}u\!\left( t \right) = {B^{ - 1}}\!\left( \begin{array}{l}- {k_2}{e_2}\!\left( t \right) - {e_1}\!\left( t \right) - {\varphi _\rho }\!\left( {t,{e_2}\!\left( t \right)} \right) - {{\hat \varepsilon }_v}\tanh \!\left( {\dfrac{{{e_2}\!\left( t \right)}}{\alpha }} \right)\\[5pt] - \hat W_a^T{\Phi _a}\!\left( {{Z_a}} \right) - \dfrac{1}{4}{e_2}\!\left( t \right) + {{\dot x}_{2c}}\!\left( t \right)\end{array} \right)\end{align}

where ${k_2}$ is the control gain.

The critic NN which can be used to appraise control performance and make feedback to the actor NN will be introduced in detail. Firstly, we define the integral penalty function of the controlled system as follows

(21) \begin{align} J\!\left( t \right) = \int_0^\infty {\!\left[ {e_1^T\!\left( \tau \right)Q{e_1}\!\left( \tau \right) + {u^T}\!\left( \tau \right)Ru\!\left( \tau \right)} \right]d\tau } \end{align}

Then, we approximate the penalty function $J\!\left( t \right)$ by designing the critic NN

(22) \begin{align}\hat J\!\left( t \right) = {\hat W_{\rm{c}}}^T{\Phi _c}\!\left( {{Z_c}} \right)\end{align}

where ${\hat W_c} \in {{\rm{R}}^{{p_2} \times n}},{{\rm{\Phi }}_c}\!\left( {{Z_c}} \right) \in {{\rm{R}}^{{p_2} \times 1}},{{\rm{\Phi }}_c}\!\left( {{Z_c}} \right) = {e^{ - \frac{{{{\!\left( {{Z_c} - \mu } \right)}^2}}}{{2{\sigma ^2}}}}},{\rm{\;\;}}{Z_c} = {e_1}\!\left( t \right)$ .

Constructing the residual mean square error function of the critic NN structure, one has

(23) \begin{align}\begin{array}{l}{e_c}\!\left( t \right) = e_1^T\!\left( t \right)Q{e_1}\!\left( t \right) + {u^T}\!\left( t \right)Ru\!\left( t \right) + \hat W_c^T\,\nabla {\Phi _c}\,{{\dot x}_1}\!\left( t \right)\\[5pt] {E_c}\!\left( t \right) = \dfrac{1}{2}e_c^T\!\left( t \right){e_c}\!\left( t \right)\end{array}\end{align}

where $\nabla {{\rm{\Phi }}_c} = \partial {{\rm{\Phi }}_c}\!\left( {{x_1}} \right)/\partial {x_1}$ and $\nabla {{\rm{\Phi }}_c} \in {{\rm{R}}^{{p_2} \times n}}$ . The update goal of the weight of the critic NN is to minimise ${E_c}\!\left( t \right)$ , thus the update rate of the critic network weight is obtained according to the gradient descent method

(24) \begin{align} {{\dot {\hat W}}_c} &= - {\Gamma _{W_c}}{\lambda _{W_c}}{e_c} - {\Gamma _{W_c}}{\lambda _{W_c}}{{\hat W}_c}\nonumber\\[5pt] &= - {\Gamma _{{W_c}}}\!\left[ {{\lambda _{{W_c}}}\!\left( {\lambda _{{W_c}}^T{{\hat W}_c} + e_1^T\!\left( t \right)Q{e_1}\!\left( t \right) + {u^T}\!\left( t \right)Ru\!\left( t \right)} \right)} \right] - {\Gamma _{{W_c}}}{\lambda _{{W_c}}}{{\hat W}_c} \end{align}

where ${\lambda _{{W_c}}} = \nabla {{\rm{\Phi }}_c}\dot x\!\left( t \right),{\rm{\;\;}}{{\rm{\Gamma }}_{{W_c}}},{\rm{\;\;}}{\lambda _{{W_c}}}\gt 0$ .

Finally, the adaptive laws of ${\dot{\hat{W}}_a},{\dot{\hat{W}}_c},{\dot {\hat {\varepsilon}}_v}$ are listed as follows

(25) \begin{align} \dot{\hat W}_a &= {\Gamma _{{W_a}}}{\Phi _a}\!\left( {{Z_a}} \right)\!\left( {e_2^T\!\left( t \right) + \hat J\Omega _{}^T} \right) - {\Gamma _{{W_a}}}{\lambda _{{W_a}}}{{\hat W}_a}\nonumber\\[5pt] \dot{\hat W}_c &= - {\Gamma _{{W_c}}}\!\left[ {{\lambda _{{W_c}}}(\lambda _{{W_c}}^T{{\hat W}_c} + e_1^TQ{e_1} + {u^T}Ru)} \right] - {\Gamma _{{W_c}}}{\lambda _{{W_c}}}{{\hat W}_c}\nonumber\\[5pt] \dot {\hat \varepsilon }_v &= e_2^T\!\left( t \right)\tanh \!\left( {\frac{{{e_2}\!\left( t \right)}}{\alpha }} \right) - {\lambda _\varepsilon }{{\hat \varepsilon }_v} \end{align}

For the sake of analysis, we define ${{\tilde{\ast}}} = {{\hat{\ast}}} - {{\ast}}$ to represent the estimation error of the unknown variable ${\rm{*}}$ .

4.0 Stability analysis

Defining ${e_0}\!\left( t \right) = \int_0^t {{e_1}\!\left( s \right)ds}$ and combining Equations (7), (8) and (20), then the closed relation can be obtained

(26) \begin{align}\begin{aligned}{{\dot e}_0}\!\left( t \right) &= {e_1}\!\left( t \right)\\[5pt] {{\dot e}_1}\!\left( t \right) &= - {k_0}{e_0}\!\left( t \right) - {k_1}{e_1}\!\left( t \right) + {e_2}\!\left( t \right) - {{\tilde d}_1}\!\left( t \right)\\[5pt] {{\dot e}_2}\!\left( t \right) &= - {k_2}{e_2}\!\left( t \right) - {e_1}\!\left( t \right) + {d_2}\!\left( t \right) + \chi \!\left( {{x_2}\!\left( t \right),\eta \!\left( t \right)} \right)\\[5pt] &- \hat W_a^T{\Phi _a}\!\left( {{Z_a}} \right) - {{\hat \varepsilon }_v}\tanh \!\left( {\frac{{{e_2}\!\left( t \right)}}{\alpha }} \right) - \frac{1}{4}{e_2}\!\left( t \right) - {\varphi _\rho }\!\left( {t,{e_2}\!\left( t \right)} \right)\end{aligned}\end{align}

The purpose of this paper is to construct an efficient controller to ensure the stability of the closed relation described in Equation (26). The stability of the straight air compound missile system with the proposed control scheme can be revealed by the following theorem.

Proof. The Lyapunov function $V$ is selected as

(27) \begin{align}V &= {V_1} + {V_2}\nonumber\\[5pt] {V_1} &= \frac{1}{2}e_0^T\!\left( t \right){e_0}\!\left( t \right) + \frac{1}{2}e_1^T\!\left( t \right){e_1}\!\left( t \right)\\[5pt] {V_2} &= \frac{1}{2}e_2^T\!\left( t \right){e_2}\!\left( t \right) + \frac{1}{2}\textrm{Tr}\!\left( {\tilde W_a^T\Gamma _{{W_a}}^{ - 1}{{\tilde W}_a}} \right) + \frac{1}{2}\textrm{Tr}\!\left( {\tilde W_c^T\Gamma _{{W_c}}^{ - 1}{{\tilde W}_c}} \right) + \frac{1}{2}{{\tilde \varepsilon }^T}_v{{\tilde \varepsilon }_v} + \frac{{r\!\left( t \right)}}{{{\Gamma _r}}}{\rm{ + }}{J^ * }\!\left( {{x_1}} \right)\nonumber\end{align}

Taking the derivative of both sides of the Equation (27), we can get that

(28) \begin{align} \dot V &= {{\dot V}_1} + {{\dot V}_2}\nonumber\\[5pt] {{\dot V}_1} &= e_0^T\!\left( t \right){{\dot e}_0}\!\left( t \right) + e_1^T\!\left( t \right){{\dot e}_1}\!\left( t \right)\\[5pt] {\dot V}_2 &= e_2^T \!\left( t \right){{\dot e}_2} \!\left( t \right) + \textrm{Tr} \!\left( {{\tilde{W}}_a^T \Gamma _{{W_a}}^{-1} {\dot{\tilde W}}_a} \right) + \textrm{Tr}\!\left( {\tilde W_c^T\Gamma _{{W_c}}^{ - 1}{\dot{\tilde W}}_c} \right) + {{\tilde \varepsilon }^T}_v {\dot {\tilde \varepsilon }_v} - \frac{\gamma _0} {\Gamma_r} r\!\left( t \right){\rm{ + }}\frac{\rho \!\left( t \right)} {\Gamma _r}{\rm{+}} J{_x^ {*T}}{{\dot x}_1}\nonumber \end{align}

Substituting Equation (26) into ${\dot V_1}$ term of Equation (28), one has

(29) \begin{align}{\dot V_1} = e_0^T\!\left( t \right){e_1}\!\left( t \right) - {k_0}e_1^T\!\left( t \right){e_0}\!\left( t \right) - {k_1}e_1^T\!\left( t \right){e_1}\!\left( t \right) + e_1^T\!\left( t \right){e_2}\!\left( t \right) - e_1^T\!\left( t \right){\tilde d_1}\!\left( t \right)\end{align}

Defining ${\bar e_1} = {\left[ {e_0^T\!\left( t \right),e_1^T\!\left( t \right)} \right]^T}$ and utilising the following inequality

(30) \begin{align}e_1^T\!\left( t \right){{\tilde d}_1}\!\left( t \right) &\le \frac{1}{2}e_1^T\!\left( t \right)e_1^{}\!\left( t \right) + \frac{1}{2}\tilde d_1^T\!\left( t \right){{\tilde d}_1}\!\left( t \right)\nonumber\\[5pt] &\le \frac{1}{2}e_1^T\!\left( t \right)e_1^{}\!\left( t \right) + \frac{1}{2}\varepsilon _d^2\end{align}

Equation (29) can be rewritten as

(31) \begin{align}\begin{array}{l}{{\dot V}_1} \le - \bar e_1^T\!\left( t \right)A{{\bar e}_1}\!\left( t \right) + e_1^T\!\left( t \right){e_2}\!\left( t \right) + \dfrac{1}{2}\varepsilon _d^2\\[5pt] A = \left[ {\begin{array}{*{20}{c}} 0&{ - 1}\\[5pt] {{k_0}}&{ - \dfrac{1}{2} + {k_1}} \end{array}} \right]\end{array}\end{align}

where we assume that $\|{\tilde d_1}\!\left( t \right)\| \le {\varepsilon _d}$ holds and ${\varepsilon _d}$ is a positive constant.

Thanks to Equation (16) and ${\dot e_2}\!\left( t \right)$ term of Equation (26), we can get the following inequality

(32) \begin{align} \begin{aligned}e_2^T\!\left( t \right){{\dot e}_2}\!\left( t \right) &\le - {k_2}e_2^T\!\left( t \right){e_2}\!\left( t \right) - e_2^T\!\left( t \right){e_1}\!\left( t \right) - e_2^T\!\left( t \right){\varphi _\rho }\!\left( {t,{e_2}\!\left( t \right)} \right) - e_2^T\!\left( t \right){{\hat \varepsilon }_v}\tanh \!\left( {\frac{{{e_2}\!\left( t \right)}}{\alpha }} \right)\\[5pt] &- e_2^T\!\left( t \right)\tilde W_a^T{\Phi _a}\!\left( {{Z_a}} \right) + e_2^T\!\left( t \right)\!\left( {{d_2}\!\left( t \right) + {\varepsilon _{{W_a}}}} \right) + \sum\limits_{i = 1}^3 {{\varepsilon _i}}\end{aligned} \end{align}

where

(33) \begin{align}&e_2^T\!\left( t \right)\!\left( {{d_2}\!\left( t \right) + {\varepsilon _{{W_a}}}} \right) - e_2^T\!\left( t \right){{\hat \varepsilon }_v}\tanh \!\left( {\frac{{{e_2}\!\left( t \right)}}{\alpha }} \right)\nonumber\\[5pt] &= e_2^T\!\left( t \right)\!\left( {D\!\left( t \right)} \right) - e_2^T\!\left( t \right){{\hat \varepsilon }_v}\tanh \!\left( {\frac{{{e_2}\!\left( t \right)}}{\alpha }} \right)\\[5pt] &\le e_2^T\!\left( t \right){\varepsilon _v}\tanh \!\left( {\frac{{{e_2}\!\left( t \right)}}{\alpha }} \right) + \kappa \alpha {\varepsilon _v} - e_2^T\!\left( t \right){{\hat \varepsilon }_v}\tanh \!\left( {\frac{{{e_2}\!\left( t \right)}}{\alpha }} \right)\nonumber\\[5pt] &= - e_2^T\!\left( t \right){{\tilde \varepsilon }_v}\tanh \!\left( {\frac{{{e_2}\!\left( t \right)}}{\alpha }} \right) + \kappa \alpha {\varepsilon _v}\nonumber\end{align}

Thus, the following inequality can be readily obtained

(34) \begin{align} e_2^T\!\left( t \right){{\dot e}_2}\!\left( t \right) &\le - {k_2}e_2^T\!\left( t \right){e_2}\!\left( t \right) - e_2^T\!\left( t \right){e_1}\!\left( t \right) - e_2^T\!\left( t \right){\varphi _\rho }\!\left( {t,{e_2}\!\left( t \right)} \right)\nonumber\\[5pt] &- e_2^T\!\left( t \right){{\tilde \varepsilon }_v}\tanh \!\left( {\frac{{{e_2}\!\left( t \right)}}{\alpha }} \right) - e_2^T\!\left( t \right)\tilde W_a^T{\Phi _a}\!\left( {{Z_a}} \right) + \sum\limits_{i = 1}^3 {{\varepsilon _i}} + \kappa \alpha {\varepsilon _v} \end{align}

Substituting Equation (34) into ${\dot V_2}$ term of Equation (28), one has

(35) \begin{align}{{\dot V}_2} &\le - {k_2}e_2^T\!\left( t \right){e_2}\!\left( t \right) - e_2^T\!\left( t \right){e_1}\!\left( t \right) - e_2^T\!\left( t \right){\varphi _\rho }\!\left( {t,{e_2}\!\left( t \right)} \right) - e_2^T\!\left( t \right){{\tilde \varepsilon }_v}\tanh \!\left( {\frac{{{e_2}\!\left( t \right)}}{\alpha }} \right) - e_2^T\!\left( t \right)\tilde W_a^T{\Phi _a}\!\left( {{Z_a}} \right)\nonumber\\[5pt] &+ \textrm{Tr}\!\left( {\tilde W_a^T \Gamma _{{W_a}}^{-1} \dot{\tilde W}_a} \right) + \textrm{Tr}\!\left( {\tilde W_c^T\Gamma _{{W_c}}^{ - 1}\dot{\tilde W}_c} \right) + {{\tilde \varepsilon }^T}_v {{\dot {\tilde \varepsilon} }_v} - \frac{{{\gamma _0}}}{{{\Gamma _r}}}r\!\left( t \right){\rm{ + }}\frac{{\rho \!\left( t \right)}}{{{\Gamma _r}}} + J{_x^ {*T}}{{\dot x}_1} + \sum\limits_{i = 1}^3 {{\varepsilon _i}} + \kappa \alpha {\varepsilon _v} \end{align}

For any vector $\xi \in {{\rm{R}}^n}$ , we define

(36) \begin{align}\mathrm{Tanh}\!\left( {\xi \!\left( t \right)} \right) = {\left[ {\tanh {\xi _1}\!\left( t \right),\tanh {\xi _2}\!\left( t \right), \cdots ,\tanh {\xi _n}\!\left( t \right)} \right]^T}\end{align}

Therefore, the following formula holds

(37) \begin{align}&\frac{{\rho \!\left( t \right)}}{{{\Gamma _r}}} = \frac{{\rho \!\left( t \right)}}{{{\Gamma _r}}}\!\left( {1 - 16\textrm{Tanh}^T \!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)\textrm{Tanh}\!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)} \right) + e_2^T\!\left( t \right){\varphi _\rho }\!\left( {t,{e_2}\!\left( t \right)} \right)\nonumber\\[5pt] &{\varphi _\rho }\!\left( {t,{e_2}\!\left( t \right)} \right) = \frac{{16{e_2}\!\left( t \right)\rho \!\left( t \right)}}{{{\Gamma _r}e_2^T\!\left( t \right){e_2}\!\left( t \right)}}\textrm{Tanh}^T \!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)\textrm{Tanh}\!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)\end{align}

Then, combining Equations (31), (35)–(37), we have

(38) \begin{align}\dot V &= {{\dot V}_1} + {{\dot V}_2}\nonumber \\[5pt] &\le - \bar e_1^T\!\left( t \right)A{{\bar e}_1}\!\left( t \right) - {k_2}e_2^T\!\left( t \right){e_2}\!\left( t \right) - e_2^T\!\left( t \right){{\tilde \varepsilon }_v}\tanh \!\left( {\frac{{{e_2}\!\left( t \right)}}{\alpha }} \right) - e_2^T\tilde W_a^T{\Phi _a}\!\left( {{Z_a}} \right)\nonumber {}\\[5pt] &+ \textrm{Tr} \!\left( {\tilde W_a^T\Gamma _{{W_a}}^{ - 1} {{\dot {\tilde W}}_a}} \right) + \textrm{Tr}\!\left( {\tilde W_c^T\Gamma _{{W_c}}^{ - 1} {{\dot {\tilde W}}_c}} \right) + {{\tilde \varepsilon }^T}_v {{\dot{\tilde \varepsilon}}_v} - \frac{{{\gamma _0}}}{{{\Gamma _r}}}r\!\left( t \right) + J{_x^{*T} }{{\dot x}_1}\\[5pt] &{{ + \rho \!\left( t \right)\!\left( {1 - 16\textrm{Tanh}^T \!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)\textrm{Tanh}\!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)} \right)} \mathord{\!\left/ {\vphantom {{ + \rho \!\left( t \right)\!\left( {1 - 16Tan{h^T}\!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)Tanh\!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)} \right)} {{\Gamma _r}}}} \right. } {{\Gamma _r}}} + \frac{1}{2}\varepsilon _d^2 + \sum\limits_{i = 1}^3 {{\varepsilon _i}} + \kappa \alpha {\varepsilon _v}\nonumber \end{align}

By using the adaptive laws in Equation (25) and considering the following inequalities

(39) \begin{align} - {\tilde \varepsilon ^T}_v {\hat \varepsilon _v} \le - \frac{1}{2}{\tilde \varepsilon ^2}_v + \frac{1}{2}\varepsilon _v^2 \end{align}

then, it can be concluded that

(40) \begin{align}\dot V &\le - \bar e_1^T\!\left( t \right)A{{\bar e}_1}\!\left( t \right) - {k_2}e_2^T\!\left( t \right){e_2}\!\left( t \right) - \frac{{{\lambda _\varepsilon }}}{2}{{\tilde \varepsilon }^2}_v + \textrm{Tr}\!\left( {\tilde W_a^T\!\left( {{\Phi _a}\hat J\Omega _{}^T - {\lambda _{{W_a}}}{{\hat W}_a}} \right)} \right)\nonumber \\[5pt] &+ \textrm{Tr}\!\left( {\tilde W_c^T\!\left( { - {\lambda _{{W_c}}}(\lambda _{{W_c}}^T{{\hat W}_c} + {\varepsilon _c}) - {\lambda _{{W_c}}}{{\hat W}_c}} \right)} \right) + J{_x^{*T}}{{\dot x}_1} - \frac{{{\gamma _0}}}{{{\Gamma _r}}}r\!\left( t \right) + \frac{1}{2}\varepsilon _d^2\\[5pt] &+ \sum\limits_{i = 1}^3 {{\varepsilon _i}} + \frac{{{\lambda _\varepsilon }}}{2}\varepsilon _v^2 + {{\rho \!\left( t \right)\!\left( {1 - 16\textrm{Tanh}^T \!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)\textrm{Tanh} \!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)} \right)} \mathord{\!\left/ {\vphantom {{\rho \!\left( t \right)\!\left( {1 - 16Tan{h^T}\!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)Tanh\!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)} \right)} {{\Gamma _r}}}} \right.} {{\Gamma _r}}}\nonumber \end{align}

Considering $Tr\!\left( {\tilde W_a^T\!\left( {{{\rm{\Phi }}_a}\hat J{{\rm{\Omega }}^T} - {\lambda _{{W_a}}}{{\hat W}_a}} \right)} \right)$ term of Equation (40), we can obtain that

(41) \begin{align}&\textrm{Tr}\!\left( {\tilde W_a^T\!\left( {{\Phi _a}\hat J\Omega _{}^T - {\lambda _{{W_a}}}{{\hat W}_a}} \right)} \right)\nonumber \\[5pt] &= \textrm{Tr}\!\left( {\tilde W_a^T{\Phi _a}\tilde W_c^T{\Phi _c}\Omega _{}^T} \right) + \textrm{Tr}\!\left( {\tilde W_a^T{\Phi _a}W_c^T{\Phi _c}\Omega _{}^T} \right) - \textrm{Tr}\!\left( {\tilde W_a^T{\lambda _{{W_a}}}{{\hat W}_a}} \right)\nonumber \\[5pt] &= \tilde W_c^T{\Phi _c}\Omega _{}^T\tilde W_a^T{\Phi _a} + {W_c}^T{\Phi _c}\Omega _{}^T{{\tilde W}_a}^T{\Phi _a} - {\lambda _{{W_a}}}\textrm{Tr}\!\left( {\tilde W_a^T{{\hat W}_a}} \right)\nonumber \\[5pt] &\le {\rho _1}\tilde W_c^T{{\tilde W}_c} + \frac{{{\lambda _{\max }}\!\left( {{\Phi _c}\Omega _{}^T{\Omega _{}}\Phi _c^T} \right)\bar \Phi _a^2}}{{4{\rho _1}}}\textrm{Tr}\!\left ( {\tilde W_a^T{{\tilde W}_a}}\right )\\[5pt] &+ {\rho _2}W_c^T{W_c} + \frac{{{\lambda _{\max }}\!\left( {{\Phi _c}\Omega _{}^T\Omega \Phi _c^T} \right)\bar \Phi _a^2}}{{4{\rho _2}}}\textrm{Tr}\!\left ( {\tilde W_a^T{{\tilde W}_a}}\right ) - \frac{{{\lambda _{{W_a}}}}}{2}\tilde W_a^T{{\tilde W}_a} + \frac{{{\lambda _{{W_a}}}}}{2}W_a^T{W_a}\nonumber \end{align}

Then, considering $\textrm{Tr}\!\left( {\tilde W_c^T\!\left( { - {\lambda _{{W_c}}}\!\left( {\lambda _{{W_c}}^T{{\hat W}_c} + {\varepsilon _c}} \right){ - \lambda_{{W_c}}}{{\hat W}_c}} \right)} \right)$ term of Equations (40), the following inequality holds

(42) \begin{align} &\textrm{Tr}\!\left( {\tilde W_c^T\!\left( { - {\lambda _{{W_c}}}(\lambda _{{W_c}}^T{{\hat W}_c} + {\varepsilon _c}) - {\lambda _{{W_c}}}{{\hat W}_c}} \right)} \right)\nonumber \\[5pt] &= \textrm{Tr}\!\left( { - \tilde W_c^T{\lambda _{{W_c}}}\lambda _{{W_c}}^T{{\tilde W}_c} - \tilde W_c^T{\lambda _{{W_c}}}{\varepsilon _c} - \tilde W_c^T{\lambda _{{W_c}}}{{\hat W}_c}} \right)\nonumber \\[5pt] &= - \tilde W_c^T{\lambda _{{W_c}}}\lambda _{{W_c}}^T{{\tilde W}_c} - \tilde W_c^T{\lambda _{{W_c}}}{\varepsilon _c} - \tilde W_c^T{\lambda _{{W_c}}}{{\hat W}_c}\\[5pt] &\le \!\left( {{\rho _{\rm{3}}} + \frac{{{\lambda _{\max }}\!\left( {{{\bar \lambda }_{{W_c}}}} \right)}}{{4{\rho _{\rm{3}}}}}} \right)\tilde W_c^T{{\tilde W}_c} + {\rho _{\rm{4}}}{\lambda _{\max }}\!\left( {{\lambda _{{W_c}}}\lambda _{{W_c}}^T} \right)\tilde W_c^T{{\tilde W}_c} + \frac{1}{{4{\rho _{\rm{4}}}}}\varepsilon _c^2 - \frac{{{\lambda _c}}}{2}\tilde W_c^T{{\tilde W}_c} + \frac{{{\lambda _c}}}{2}W_c^T{W_c}\nonumber \end{align}

Moreover, the $J_x^{{\rm{*}}T}{\dot x_1}$ term of Equations (28) satisfies

(43) \begin{align}J{_x^ {*T}}{\dot x_1} \le - {\lambda _{\min }}\{ Q\} ||{e_1}|{|^2} - {\lambda _{\min }}\{ R\} ||u|{|^2}\end{align}

Substituting Equations (41)–(43) into Equation (40), we can get that

(44) \begin{align}\dot V &\le - \bar e_1^T\!\left( t \right)A{{\bar e}_1}\!\left( t \right) - {k_2}e_2^T\!\left( t \right){e_2}\!\left( t \right) - \frac{{{\lambda _\varepsilon }}}{2}{{\tilde \varepsilon }^2}_v - \frac{{{\gamma _0}}}{{{\Gamma _r}}}r\!\left( t \right) + {{\rho \!\left( t \right)\!\left( {1 - 16\textrm{Tanh}^T \!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)\textrm{Tanh} \!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)} \right)} \mathord{\!\left/ {\vphantom {{\rho \!\left( t \right)\!\left( {1 - 16\textrm{Tanh}^T \!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)Tanh\!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)} \right)} {{\Gamma _r}}}} \right.} {{\Gamma _r}}}\nonumber \\[5pt] &- \!\left( {\frac{{{\lambda _{{W_a}}}}}{2} - \frac{{{\lambda _{\max }}\!\left( {{\Phi _c}\Omega _{}^T{\Omega _{}}\Phi _c^T} \right)\bar \Phi _a^2}}{{4{\rho _1}}} - \frac{{{\lambda _{\max }}\!\left( {{\Phi _c}\Omega _{}^T\Omega \Phi _c^T} \right)\bar \Phi _a^2}}{{4{\rho _2}}}} \right)\textrm{Tr} \!\left({\tilde W_a^T{{\tilde W}_a}}\right)\nonumber \\[5pt] &- \!\left( {\frac{{{\lambda _{{W_c}}}}}{2} - {\rho _1} - {\rho _{\rm{3}}} - \frac{{{\lambda _{\max }}\!\left( {{{\bar \lambda }_{{W_c}}}} \right)}}{{4{\rho _{\rm{3}}}}} - {\rho _{\rm{4}}}{\lambda _{\max }}\!\left( {{\lambda _{{W_c}}}\lambda _{{W_c}}^T} \right)} \right)\tilde W_c^T{{\tilde W}_c} - {\lambda _{\min }}\{ Q\} ||{e_1}|{|^2} - {\lambda _{\min }}\{ R\} ||u|{|^2}\\[5pt] &+ \frac{{{\lambda _\varepsilon }}}{2}\varepsilon _v^2 + \frac{1}{2}\varepsilon _d^2 + \sum\limits_{i = 1}^3 {{\varepsilon _i}} + \frac{{{\lambda _{{W_a}}}}}{2}W_a^T{W_a} + \frac{{2{\rho _2} + {\lambda _{{W_c}}}}}{2}W_c^T{W_c} + \frac{1}{{4{\rho _{\rm{4}}}}}\varepsilon _c^2\nonumber \end{align}

Defining

(45) \begin{align} &\gamma = \min \!\left\{ \begin{array}{l}2{\lambda _{\min }}\!\left( A \right),2{k_2},2{\lambda _{\min }}\!\left( Q \right),2{\lambda _{\min }}\!\left( R \right),\;{\lambda _\varepsilon },\nonumber \\[5pt] \dfrac{{{\lambda _{{W_a}}}}}{2} - \dfrac{{{\lambda _{\max }}\!\left( {{\Phi _c}\Omega _{}^T{\Omega _{}}\Phi _c^T} \right)\bar \Phi _a^2}}{{4{\rho _1}}} - \dfrac{{{\lambda _{\max }}\!\left( {{\Phi _c}\Omega _{}^T\Omega \Phi _c^T} \right)\bar \Phi _a^2}}{{4{\rho _2}}},\nonumber \\[5pt] \dfrac{{{\lambda _c}}}{2} - {\rho _1} - {\rho _{\rm{3}}} - \dfrac{{{\lambda _{\max }}\!\left( {{{\bar \lambda }_{{W_c}}}} \right)}}{{4{\rho _{\rm{3}}}}} - {\rho _{\rm{4}}}{\lambda _{\max }}\!\left( {{\lambda _{{W_c}}}\lambda _{{W_c}}^T} \right),\end{array} \right\}\nonumber \\[5pt] &{\varepsilon _f} = - \dfrac{{{\gamma _0}}}{{{\Gamma _r}}}r\!\left( t \right) + \dfrac{{{\lambda _\varepsilon }}}{2}\varepsilon _v^2 + \dfrac{1}{2}\varepsilon _d^2 + \sum\limits_{i = 1}^3 {{\varepsilon _i}} + \dfrac{{{\lambda _{{W_a}}}}}{2}W_a^T{W_a} + \dfrac{{2{\rho _2} + {\lambda _{{W_c}}}}}{2}W_c^T{W_c} + \dfrac{1}{{4{\rho _{\rm{4}}}}}\varepsilon _c^2 \end{align}

Thanks to Equation (45), we can obtain that

(46) \begin{align}\dot V \le - \gamma V + {\varepsilon _f}{{ + \rho \!\left( t \right)\!\left( {1 - 16\textrm{Tanh}^T\!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)\textrm{Tanh}\!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)} \right)} \mathord{\!\left/ {\vphantom {{ + \rho \!\left( t \right)\!\left( {1 - 16Tan{h^T}\!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)Tanh\!\left({\frac{{e_2}\!\left( t \right)}{{{\varepsilon _\rho }}}}\right)} \right)} {{\Gamma _r}}}} \right. } {{\Gamma _r}}}\end{align}

According to Equation (46), it can be seen that signals $\left[ {{e_0}\!\left( t \right),{e_1}\!\left( t \right),{e_2}\!\left( t \right),{{\tilde \varepsilon }^{}}_v\!\left( t \right),{{\tilde W}_a}\!\left( t \right),{{\tilde W}_c}\!\left( t \right)} \right]$ are all stable and bounded. Therefore, the stability of the closed-loop system and the boundedness of all signals can be verified. The proof is complete.

5.0 Simulation study

In this section, some numerical simulations are performed to demonstrate the effectiveness and performance of the proposed STDO-based adaptive reinforcement learning (ARL) control method. To show the advantages of the proposed STDO-ARL method, the STDO-ARL without STDO and the STDO-ARL without reinforcement learning (RL) are also considered for comparison, as shown in Figs. 2 and 3. On the other hand, the robustness of the proposed STDO-ARL method is reflected by several external disturbances ${d_1}\!\left( t \right)$ and uncertainties $\chi \!\left( t \right)$ of different degrees imposed on the system as listed in Table 1, and the results are shown in Figs. 7 and 8.

Figure 2. Comparison chart of the tracking performance of $\alpha $ under different methods.

Figure 3. Comparison chart of the tracking performance of $\beta $ under different methods.

Table 1. Model parameter values in different cases

The initial values of the system for simulation are listed as follows: ${x_1} = {\left[ {\begin{array}{c}{0.0675}\;\;\;\; { - 0.5738}\end{array}} \right]^T},{x_2} = {\left[ {\begin{array}{c}0\;\;\;\; 0\end{array}} \right]^T}$ ; the weights of the actor network and the critic network are respectively set as: ${\hat W_a} = \textrm{zeros}\!\left( {22,1} \right),{\hat W_c} = \textrm{zeros}\!\left( {11,1} \right)$ ; the mismatched disturbance of the system is ${\hat d_1} = {\left[ {\begin{array}{c@{\quad}c} 0&0 \end{array}} \right]^T}$ ; what’s more, ${P_2} = {\left[ {\begin{array}{c@{\quad}c}0 & 0\end{array}} \right]^T}$ and ${\hat x_2} = {\left[ {\begin{array}{c@{\quad}c}0&0 \end{array}} \right]^T}$ .

We choose the unmodeled dynamics as $\eta = 1$ and the dynamic auxiliary signal as $r = 2$ . The mismatched disturbance ${d_1}\!\left( t \right)$ in the simulation are set as two different trapezoidal waves: ${d_1}\!\left( t \right) = \left[ {\begin{array}{c}{D\!\left( {t,5,3} \right)}\\[2pt] {D\!\left( {t,5,1} \right)}\end{array}} \right]$ . The uncertainties that are affected by the unmodeled dynamics are supposed to be $\chi \!\left( t \right) = 0.5{x_1}\!\left( t \right){\rm{sin}}\!\left( t \right) + \eta {x_2}\!\left( t \right)$ . In this control method, the system matrix is selected as $B = \left[ {\begin{array}{c@{\quad}c@{\quad}c@{\quad}c}1 &0 &1& 0\\[2pt] 0& 1 &0 & 1\end{array}} \right]$ and other control constants are set as ${{\rm{\Gamma }}_r} = 120,{\rm{\;\;}}{\varepsilon _\rho } = 0.1$ . The control gains are designed as ${k_0} = 3,{\rm{\;\;}}{k_1} = {k_2} = 6$ . The control parameters of STDO disturbance observer are designed as ${K_{{d_1}}} = 5,{\rm{\;\;}}{K_{{P_1}}} = 2,{\rm{\;\;}}{K_{{P_2}}} = 0.1$ . And the adaptive control parameters of reinforcement learning actor network and critic network are set as ${{\rm{\Gamma }}_{{W_a}}} = 0.2,{\rm{\;\;}}{\lambda _{{W_a}}} = 2.5,{\rm{\;\;\Omega }} = {\left[ {\begin{array}{c@{\quad}c}2 & 1\end{array}} \right]^T},{\rm{\;\;}}{{\rm{\Gamma }}_{{W_c}}} = 0.2,{\rm{\;\;}}{\lambda _{{W_c}}} = 2.5,{\rm{\;\;}}Q = diag\!\left( {\left[ {1,{\rm{\;\;}}1} \right]} \right),{\rm{\;\;}}R = diag\!\left( {\left[ {2,{\rm{\;}}1,{\rm{\;}}0,{\rm{\;}}1} \right]} \right)$ . According to the above simulation parameters, the following simulations are carried out: the comparison simulation of different methods and different cases.

5.1 Simulation comparison under different methods

This part is a comparative simulation under different methods. According to Figs. 2 and 3, it is obvious that for the time-varying desired signal, the proposed STDO-ARL control scheme can achieve satisfactory results for the tracking control problems of the straight air compound missile with external disturbances and unmodeled dynamics. While the tracking performance of the proposed method without STDO and the proposed method without RL is not ideal, it may produce undesired tracking errors and cannot ensure the tracking accuracy.

Moreover, according to Figs. 4, 5 and 6, all signals in the closed-loop control system are bounded during the whole control process by using the proposed STDO-ARL method. In summary, the proposed STDO-ARL control method under unmodeled dynamics and disturbances can achieve satisfactory control performance.

Figure 4. The trajectories of the adaptive parameters of the proposed STDO-ARL scheme.

Figure 5. Disturbance estimation effect of $\hat{d}_1 \left(t\right)$ based on STDO.

Figure 6. Variation diagram of the weight norm of the reinforcement learning actor NN $\left\| {{{\hat W}_a}} \right\|$ and the critic NN $\left\| {{{\hat W}_c}} \right\|$ .

5.2 Simulation comparison under different cases

This part is a comparative simulation for the proposed STDO-ARL control scheme under different cases, as shown in Figs. 7 and 8. From the analysis of the simulation results, the proposed STDO-ARL method has the characteristics of high anti-disturbance in dealing with trapezoidal signals and sine-cosine combined signals. However, there will be slight fluctuations when dealing with square-wave signal disturbance. Moreover, for various complex uncertainty conditions, the reinforcement learning structure can be used to fit them effectively. To sum up, the proposed STDO-ARL method has strong robustness and anti-disturbance ability under different cases.

Figure 7. Comparison chart of the tracking performance of $\alpha $ under different cases.

Figure 8. Comparison chart of the tracking performance of $\beta $ under different cases.