1. Introduction

The work concerning qualitative properties of solutions was extensively investigated by many researchers. Fruitful results have been obtained on the existence, non-existence, symmetry and regularity and so on. Berestycki and Nirenberg [Reference Berestycki and Nirenberg3] obtained the monotonicity for the unbounded positive solutions of elliptic equations in the case of a ‘coercive’ Lipschitz graph. In [Reference Berestycki, Caffarelli and Nirenberg6], Berestycki et al. proved the monotonicity and uniqueness for elliptic equations in unbounded Lipschitz domains. Angenent [Reference Angenent1] and Clément and Sweers [Reference Clément and Sweers15] derived that a bounded positive solution of elliptic equations only depends on the $x_n$-variable in a upper half space. Chen et al. [Reference Chen, Li and Li12] worked on the symmetry and non-existence of positive solutions of equations with fractional Laplacian in different types of domains. For more related results, please see [Reference Cabré and Tan7, Reference Chen, Fang and Yang10, Reference Chen and Li11, Reference Chen and Li14, Reference Gidas, Ni and Nirenberg18], and the references therein.

In this work, we investigate the following system involving the fractional Laplacian:

(1.1)\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s} u (x)= f(u,\,v), \\ (-\Delta)^{s} v (x)= g(u,\,v) \end{array} \right. \end{equation}

in a bounded domain and an unbounded domain, respectively.

From an applicable point of view, the fractional Laplacian have caught researchers’ attention because of its non-locality and its applications in physical sciences. So far it has been utilized to model diverse physical phenomena, such as anomalous diffusion and quasi-geostrophic flows, turbulence and water waves, molecular dynamics and relativistic quantum mechanics of stars (see [Reference Bouchard and Georges5, Reference Caffarelli and Vasseur9, Reference Constantin16, Reference Tarasov and Zaslasvky22] and the references therein). It also has various applications in probability and finance [Reference Applebaum2, Reference Bertoin4]. In particular, the fractional Laplacian can be understood as the infinitesimal generator of a stable Lévy diffusion process [Reference Bertoin4].

The fractional Laplacian is a non-local pseudo-differential operator, taking the form

(1.2)\begin{equation} (-\Delta)^{s} u(x) = C_{n,\alpha} \, PV \int_{R^n} \frac{u(x)-u(z)}{|x-z|^{n+2s}} \,{\rm d}z,\end{equation}

where $s\in (0,\,1)$ and PV stands for the Cauchy principal value. This operator is well defined for $u\in C^{\infty }_0(R^n)$. In this space, it can also be defined equivalently in terms of the Fourier transform

\[ (-\Delta)^{s} u (x) = {\mathcal{F}}^{{-}1}(|\xi|^{2s} {\mathcal{F}}u(\xi))(x), \]

where $\mathcal {F}$ is the Fourier transform, and ${\mathcal {F}}^{-1}$ is the inverse Fourier transform. The fractional Laplacian can be extended to locally integrable functions with certain growth control—the weighted $L^1$-space:

\[ \mathcal{L}_{2s}=\{u: R^n\rightarrow R \mid \int_{R^n}\frac{|u(x)|}{1+|x|^{n+2s}} \, d x <\infty\} \ \mbox{(see [21])}. \]

For $u\in \mathcal {L}_{2s}$, we define $(-\Delta )^{s} u$ as a distribution:

\[ ((-\Delta)^{s}u)( \phi) = \int_{R^n} u(x) (-\Delta)^{s} \phi(x)\,{\rm d}x, \quad \forall\ \phi \in {C^{\infty}_c}(R^n). \]

To investigate the properties of solutions of equations involving the fractional Laplacian, the method of moving planes and extension method [Reference Caffarelli and Silvestre8] have been powerful tools. In [Reference Zhuo, Chen, Cui and Yuan25], the authors employed the method of moving planes in integral forms (see [Reference Chen, Fang and Yang10, Reference Chen and Li11, Reference Chen, Li and Ou13] and the references therein) to study the symmetry of solutions. However, this method needs to establish the equivalence between the differential equations and the integral equations via Green's functions, and it is a challenge work. Also, this method depends heavily on some special properties of the corresponding Green's functions. So far, there are few results about the Green's functions in general domains. Chen et al. [Reference Chen, Li and Li12] developed a direct method of moving planes for the fractional Laplacian. Based on the classical sliding method for Laplacian [Reference Berestycki and Nirenberg3], Wu and Chen [Reference Wu and Chen23, Reference Wu and Chen24] developed the direct sliding method for the scalar equations involving the fractional Laplacian. Actually, the direct sliding method does not depend on the Green's functions. In this paper, we employ the direct sliding method [Reference Wu and Chen23, Reference Wu and Chen24] to derive the monotonicity of solutions of system (1.1) in a general bounded domain and an infinite cylinder.

The analogue problem to (1.1) for the fractional Laplacian has been investigated by many authors:

(1.3)\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s} u_i (x)= f_i(u_1,\dots,u_m), \ x\in \Omega, \ & i=1,\dots,m, \\ u_i(x)=0, \ x\in {R}^n\setminus \Omega, \ & i=1,\dots,m. \end{array} \right. \end{equation}

In the case $\Omega$ be a unit ball or half space, Mou [Reference Mou19] proved the symmetry and monotonicity of positive solutions of (1.3) by the integral equation approach. When $i=1,\,2$, $f_1=u^p_2$, $f_2=u^q_1$ and $\Omega =R^n_+$, Quaas and Xia [Reference Quaas and Xia20] obtained the non-existence of positive solutions of (1.3) by the method of moving planes with an improved Aleksandrov–Bakelman–Pucci type estimate for the fractional Laplacian.

In this paper, we consider the non-linear equations involving the fractional Laplacian in general domains. Due to the non-local nature of the operator, we need to set exterior conditions in domain $\Gamma$,

\[ u(x)=\varphi(x),\quad v(x)=\psi(x), \quad x\in \Gamma^c. \]

In order to ensure the monotonicity of solutions, one has to impose the necessary exterior conditions (P) on $\Gamma$: For any three points $x^1=(x',\,x^1_n)$, $x^2=(x',\,x^2_n)$ and $x^3=(x',\,x^3_n)$ with $x^1_n< x^2_n< x^3_n$, $x'\in {R}^{n-1}$, and $x^1$, $x^3\in \Gamma ^c$, $x^1$, $x^2$ and $x^3$ satisfy

\begin{align*} & \varphi(x^1)< u(x^2)<\varphi(x^3), \quad \mbox{for }x^2\in \Gamma,\\ & \varphi(x^1)\leq\varphi(x^2)\leq\varphi(x^3),\ \mbox{for }x^2\in \Gamma^c,\\ & \psi(x^1)< v(x^2)<\psi(x^3),\ \mbox{for }x^2\in \Gamma,\\ & \psi(x^1)\leq\psi(x^2)\leq\psi(x^3),\ \mbox{for }x^2\in \Gamma^c. \end{align*}

We first study (1.1) in bounded domain $G$, and establish monotonicity of solutions for (1.1) as following:

Theorem 1.1 Let $u$, $v\in \mathcal {L}_{2s}\cap C^{1,1}_{loc}(G)\cap C(\bar {G})$ and $(u,\,v)$ be a pair of solution for

(1.4)\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s} u (x)= f(u,\,v), \ & x\in G, \\ (-\Delta)^{s} v (x)= g(u,\,v), \ & x\in G, \\ u(x)=\varphi(x),\ v(x)=\psi(x),\ & x\in G^c, \end{array} \right. \end{equation}

where $G$ is a convex bounded domain in $x_n$ direction, and $u$, $v$ satisfy the exterior condition $(\mathbf{P})$ on $G$. If $\frac {\partial f}{\partial v}>0$, $\frac {\partial g}{\partial u}>0$ for $x\in G$, and $\frac {\partial f}{\partial u}$, $\frac {\partial f}{\partial v}$, $\frac {\partial g}{\partial u}$, $\frac {\partial g}{\partial v}$ are bounded from above in $G$, then $u$ and $v$ are monotone increasing with respect to $x_n$-variable in $G$. More precisely, for any $\tau >0$, one has

\[ u(x',\tau+x_n)>u(x',x_n),\quad \forall\ (x',\tau+x_n), \,(x',x_n)\in G. \]

In the proof of theorem 1.1, employing the argument by contradiction at extreme points, we first derive the key tool—narrow region principle in bounded domains. Combining with the sliding method, the solutions of system (1.4) are proved to be monotonic in the bounded domain $G$. Next we briefly introduce the sliding method. For any positive real number $\tau$, by sliding downward $\tau$ units from the bounded domain $G$, we have

\[ G_{\tau}=\{x-\tau e_n\,|\,x\in G\}, \]

here $e_n=(0,\,\dots,\,0,\,1)$. Denote

\begin{align*} & u^{\tau}(x)=u(x',x_n+\tau),\quad v^{\tau}(x)=v(x',x_n+\tau),\\ & \tilde{U}^{\tau}(x)=u^{\tau}(x)-u(x),\quad \tilde{V}^{\tau}(x)=v^{\tau}(x)-v(x). \end{align*}

For $\tau$ sufficiently close to the width of $G$ in $x_n$ direction, it is easy to see that $G\cap G_{\tau }$ is a narrow region. Applying the narrow region principle in bounded domains yields that

(1.5)\begin{equation} \tilde{U}^{\tau}(x)\geq 0,\quad \tilde{V}^{\tau}(x)\geq 0,\quad x\in G\cap G_{\tau}.\end{equation}

Note that (1.5) provides a starting position to slide the domain $G_{\tau }$. Then we slide $G_{\tau }$ back upward as long as inequality (1.5) holds to its limiting position. In fact, the domain should be slid to $\tau =0$. We conclude that the solutions of (1.4) are monotone increasing in $x_n$ variable.

Considering the unbounded domain $\Omega$, because of the unboundedness property for $\Omega$, the extremum points in $\Omega$ cannot be attained which makes it hard to apply the narrow region principle in unbounded domains directly. To overcome this difficulty, we first take a minimization sequence approaching to the infimum, and then make some perturbation about the sequence to attain the extremum points in some bounded domain. Combining with the iteration method, we can deduce the narrow region principle in unbounded domains. Then the monotonicity of solutions for (1.1) in an unbounded domain $\Omega$ is obtained with the aid of the direct sliding method.

Theorem 1.2 Let $u$, $v\in \mathcal {L}_{2s}\cap C^{1,1}_{loc}(R^n)$ be a pair of solution for

(1.6)\begin{equation} \left\{\begin{array}{@{}ll} (-\Delta)^{s} u (x)= f(u,\,v), \ & x\in \Omega, \\ (-\Delta)^{s} v (x)= g(u,\,v), \ & x\in \Omega, \\ u(x)=\varphi(x),\ v(x)=\psi(x),\ & x\in \Omega^c, \end{array} \right. \end{equation}

where $\Omega =\{x=(x',\,x_n)\in R^n\,|\,0< x_n< M\}$, $x'=(x_1,\,x_2,\,\dots,\,x_{n-1})$, and $M$ is a finite positive real number. $u,\,v$ satisfy the exterior condition $(\mathbf{P})$ on $\Omega$, and $u(x',\,\cdot )$ and $v(x',\,\cdot )$ are bounded with $x'\in R^{n-1}$. Suppose that $\frac {\partial f}{\partial v}>0$, $\frac {\partial g}{\partial u}>0$ for $x\in \Omega$, and $\frac {\partial f}{\partial u}$, $\frac {\partial f}{\partial v}$, $\frac {\partial g}{\partial u}$, $\frac {\partial g}{\partial v}$ are bounded from above in $\Omega$. Then $u(x)$ and $v(x)$ are monotone increasing in $x_n$-variable, that is, for any $\tau >0$,

\[ u(x',\tau+x_n)>u(x',x_n),\quad \forall (x',\tau+x_n), \,(x',x_n)\in \Omega. \]

One of the interesting point about the monotonicity of solutions is that it helps to pave the way for deriving existence, non-existence and some Sobolev inequalities, as can be seen in [Reference Chen, Li and Ou13, Reference Chen and Li14, Reference Dipierro, Soave and Valdinoci17] and the references therein.

This paper is organized as follows. In § 2, we establish some lemmas, such as the narrow region principle in bounded domains and so on. In § 3 and 4, combining the lemmas in § 2 with the sliding method, we derive the monotonicity of solutions for (1.1) on bounded domains and unbounded domains.