Lemma 2.1 Identity (2.3) holds for all functions

\[ w\in W^{1, p}_{loc}(\Omega;\mathbb{R}^n),\quad a\in W^{1,\bar p}_{loc}(\Omega;\mathbb{R}^n), \quad \zeta\in C_c^1(\Omega;\mathbb{R}^n), \]

where $n-1\le p\le \infty$ and $\bar p={p}/({p+1-n})$ are given numbers.

Proof. Note that if $w\in W^{1, p}_{loc}(\Omega ;\mathbb {R}^n)$ then $\operatorname {cof} Dw\in L^{{p}/({n-1})}_{loc}(\Omega ;\mathbb {M}^{n\times n}).$ Since $\bar p$ and ${p}/({n-1})$ are Hölder conjugate numbers, it follows, by the standard approximation arguments, that identity (2.3) holds for the given functions.

Lemma 2.2 Let $H(t)=th(t)-\int _0^t h(s)\,{\rm d}s.$ Then

(2.4)\begin{equation} \int_\Omega h(\det Dw) \operatorname{cof} Dw :D((Dw) \phi) = \int_\Omega H(\det Dw) \operatorname{div}\phi \end{equation}

holds for all $w\in C^2(\Omega ;\mathbb {R}^n)$ and $\phi \in C_c^1(\Omega ;\mathbb {R}^n).$

Proof. First we assume $h\in C^1(\mathbb {R})$ and let $a (x)=h(\det Dw(x)).$ Then $a\in C^1(\Omega )$ and $Da=h'(\det Dw ) D(\det Dw ).$ Take $\zeta =(Dw)\phi \in C_c^1(\Omega ;\mathbb {R}^n)$ in (2.3) and, from $Dw^T \operatorname {cof} Dw=(\det Dw)I$ and $H'(t)=th'(t),$ we have

\begin{align*} & \int_\Omega h(\det Dw) \operatorname{cof} Dw :D((Dw) \phi) \\ & \quad ={-}\int_\Omega h'(\det Dw ) (\operatorname{cof} Dw )D(\det Dw ) \cdot (Dw)\phi\\ & \quad ={-}\int_\Omega h'(\det Dw) (\det Dw) D(\det Dw ) \cdot \phi \\ & \quad ={-}\int_\Omega H'(\det Dw) D(\det Dw ) \cdot \phi \\ & \quad ={-}\int_\Omega D(H(\det Dw) ) \cdot \phi = \int_\Omega H(\det Dw) \operatorname{div} \phi. \end{align*}

This proves identity (2.4) for $h\in C^1(\mathbb {R}).$ The proof of (2.4) for $h\in C(\mathbb {R})$ follows by approximating $h$ by $C^1$ functions.

Proposition 2.3 Let $u\in C^2(\Omega ;\mathbb {R}^n)$ be a weak solution of (2.1). Then $h(\det Du)$ is constant on $\Omega.$

Proof. Let $d(x)=\det Du(x);$ then $d\in C^1(\Omega ).$ By (2.2) and (2.4), it follows that $\int _\Omega H(d(x))\operatorname {div}\phi \,{\rm d}x=0$ for all $\phi \in C_c^1(\Omega ;\mathbb {R}^n);$ hence $H(d(x))=C$ is a constant on $\Omega.$ Suppose, on the contrary, that $h(d(x_1))\ne h(d(x_2))$ for some $x_1,\,x_2\in \Omega.$ Then $d_1=d(x_1)\ne d_2=d(x_2),$ say $d_1< d_2.$ Since $d(x)$ is continuous, by the intermediate value theorem, it follows that $H(t)=C$ for all $t\in [d_1,\,d_2].$ Solving for $h(t)$ from the integral equation $th(t)-\int _0^t h(s)\,{\rm d}s=C$ on $(d_1,\,d_2)$, we obtain that $h(t)$ is constant on $(d_1,\,d_2),$ contradicting $h(d_1)\ne h(d_2).$

Proposition 2.4 Let $p\ge n-1$, $\bar p={p}/({p+1-n}),$ and $u\in W^{1, p}_{loc}(\Omega ;\mathbb {R}^n)$ be a weak solution of (2.1) such that $h(\det Du)\in W^{1,\bar p}_{loc}(\Omega ).$ Then $h(\det Du)$ is constant almost everywhere on $\Omega.$

Proof. With $a=h(\det Du)\in W^{1,\bar p}_{loc}(\Omega )$ and $w=u\in W^{1, p}_{loc}(\Omega ;\mathbb {R}^n)$ in lemma 2.1 and by (2.2), we have

\[ 0= \int_\Omega h(\det Du)\operatorname{cof} Du:D \zeta ={-}\int_\Omega (\operatorname{cof} Du)D(h(\det Du)) \cdot \zeta \]

for all $\zeta \in C_c^1(\Omega ;\mathbb {R}^n).$ As a result, we have $(\operatorname {cof} Du)D(h(\det Du ))=0$ a.e. on $\Omega.$ Thus $D(h(\det Du ))=0$ a.e. on the set $\Omega _0=\{x\in \Omega :\det Du(x)\ne 0\}.$ Clearly, $D(h(\det Du ))=0$ a.e. on the set $E=\{x\in \Omega :\det Du(x) = 0\}$ because $h(\det Du)=h(0)$ a.e. on $E.$ Therefore, $D(h(\det Du ))=0$ a.e. on the whole domain $\Omega ;$ this proves that $h(\det Du)$ is constant a.e. on $\Omega.$

Theorem 2.5 Let $u\in C^1(\Omega ;\mathbb {R}^2)$ be a weak solution of equation (2.1). Then $h(\det Du)$ is constant on $\Omega.$

Proof. Let $E=\{x\in \Omega :\det Du(x)=0\},$ which is relatively closed in $\Omega.$ There is nothing to prove if $E=\Omega ;$ thus we assume $\Omega _0=\Omega \setminus E\ne \emptyset.$ Let $C$ be a component of the open set $\Omega _0.$ Without loss of generality, we assume $\det Du>0$ on $C.$ Let $x_0\in C$ and $y_0=u(x_0).$ Since $y_0$ is a regular value of $u$, by the inverse function theorem, there exists an open disc $D=B_\epsilon (y_0)$ such that

\[ \begin{cases} U=u^{{-}1}(D)\ \mbox{is a subdomain of }C;\\ \det Du>0\ \mbox{on} \ U;\\ u:U\to D\ \mbox{is bijective with inverse} \ v=u^{{-}1}:D\to U;\\ v\in C^1(D;\mathbb{R}^n). \end{cases} \]

Given any $\phi \in C^1_c(D;\mathbb {R}^n),$ the function $\zeta (x)=\phi (u(x))\in C^1_c(U;\mathbb {R}^n)$ is a test function for (2.2) with $D\zeta (x)=D\phi (u(x))Du(x);$ thus, by the change of variables, we obtain that

\begin{align*} 0& =\int_\Omega h(\det Du)\operatorname{cof} Du : D\zeta \,{\rm d}x\\ & =\int_U h(\det Du(x))\operatorname{cof} Du(x) : D\phi(u(x))Du(x) \, {\rm d}x \\ & =\int_U h(\det Du(x))\det Du(x) \operatorname{tr}(D\phi(u(x))) \, {\rm d}x \\ & =\int_D h(\det Du(v(y))\operatorname{div} \phi(y) \,{\rm d}y. \end{align*}

This holding for all $\phi \in C^1_c(D;\mathbb {R}^n)$ proves that $h(\det Du(v(y))$ is constant on $D;$ hence $h(\det Du)$ is constant on $U.$ Since $C$ is connected and $h$ is continuous, it follows that $h(\det Du)$ is constant on the relative closure $\bar C$ of $C$ in $\Omega.$ If $E=\emptyset,$ we have $C=\Omega ;$ hence, $h(\det Du)$ is constant on $\Omega.$ If $E\ne \emptyset,$ we have $\bar C\cap E\ne \emptyset$ and thus $h(\det Du)=h(0)$ on $\bar C,$ which proves that $h(\det Du) =h(0)$ on $\Omega _0;$ hence $h(\det Du) =h(0)$ on the whole $\Omega.$

Corollary 2.6 Let $u$ be a weak solution of equation (2.1). Then $\det Du$ is constant almost everywhere on $\Omega$ if one of the following assumptions holds:

1. (a) $u \in C^1(\Omega ;\mathbb {R}^n)$ and $h$ is not constant on any intervals.

2. (b) $u\in W^{1,p}_{loc}(\Omega ;\mathbb {R}^n)$, $h(\det Du)\in W^{1,\bar p}_{loc}(\Omega )$, where $n-1\le p\le \infty$ and $\bar p={p}/({p+1-n})$ are given numbers, and $h$ is one-to-one.

Proof. Assuming (a), by theorem 2.5, we have that $h(\det Du)=c$ is constant in $\Omega$ and thus $\det Du(x)\in h^{-1}(c).$ Since $\Omega$ is connected, $\det Du$ is continuous in $\Omega$ and $h^{-1}(c)$ contains no intervals, it follows that $\det Du$ is constant in $\Omega.$ Assuming (b), by proposition 2.4, we have that $h(\det Du)$ is constant a.e. on $\Omega.$ Since $h$ is one-to-one, it follows that $\det Du$ is constant a.e. on $\Omega.$

Proposition 2.7 Assume $h$ is nondecreasing and let $f(t)=\int _0^t h(s)\,{\rm d}s.$ Suppose that $u\in W^{1,n}_{loc}(\Omega ;\mathbb {R}^n)$ is a weak solution of (2.1) such that $h(\det Du)$ is constant a.e. on $\Omega.$ Then for all subdomains $G\subset \subset \Omega,$ the inequality

\[ \int_G f(\det Du(x))\,{\rm d}x \le \int_G f(\det Dv(x))\,{\rm d}x \]

holds for all $v\in W^{1,n}(G;\mathbb {R}^n)$ satisfying $v-u\in W^{1,n}_0(G;\mathbb {R}^n).$

Proof. Let $h(\det Du)=\mu$ be a constant a.e. on $\Omega.$ Since $f'=h$ is nondecreasing, it follows that $f$ is convex and thus $f(t)\ge f(t_0)+h(t_0)(t-t_0)$ for all $t,\, t_0\in \mathbb {R}.$ Let $v\in W^{1,n}(G;\mathbb {R}^n)$ satisfy $v-u\in W^{1,n}_0(G;\mathbb {R}^n).$ Then

\[ f(\det Dv)\ge f(\det Du) +\mu (\det Dv-\det Du) \quad \mbox{a.e. on}\ \Omega. \]

Integrating over $G,$ since $\int _G \det Dv=\int _G \det Du$, we have

\[ \int_G f(\det Dv ) \ge \int_G f(\det Du ) +\mu \int_G (\det Dv-\det Du) =\int_G f(\det Du). \]

Theorem 2.8 Assume $h$ is nondecreasing. Let $p>n$, $\tilde p={p}/({p-n}),$ and let $u\in W^{1,p}(\Omega ;\mathbb {R}^n)$ be a weak solution of (2.1) with $h(\det Du)\in L^{\tilde p}(\Omega )$ such that

(2.6)\begin{equation} \begin{cases} u \mbox{ is one-to-one on } \Omega,\\ u(\Omega) \ \mbox{is a domain in} \ \mathbb{R}^n,\\ v =u^{{-}1}\in C(u(\Omega);\mathbb{R}^n)\ \mbox{satisfies the } {Luzin} N-property: \\ |v(B)|=0\ \mbox{for all} \ B\subset u(\Omega)\ \mbox{with} \ |B|=0. \end{cases} \end{equation}

Then $h(\det Du)$ is a constant a.e. on $\Omega.$

Proof. Recall that $\operatorname {div}(\operatorname {cof} (Du))=0.$ Without loss of generality, we assume ${h(0)=0};$ thus $h(t)t=|h(t)t|\ge 0$ for all $t\in \mathbb {R}.$ From the assumption, we have

\[ h(\det Du)\operatorname{cof} Du\in L^{p'}(\Omega;\mathbb{M}^{n\times n});\quad p'= {p}/(p-1). \]

Let $\phi \in C^1_c(u(\Omega );\mathbb {R}^n).$ Then $\zeta (x)=\phi (u(x))\in W^{1,p}_0(\Omega ;\mathbb {R}^n)$ is a legitimate test function for (2.2). Since $u$ is one-to-one on $\Omega,$ we have $N(u|\Omega ;y) =1$ for all $y\in u(\Omega );$ thus, by the change of variable formula (2.5), we obtain

\begin{align*} 0& =\int_\Omega h(\det Du)\operatorname{cof} Du : D\zeta \,{\rm d}x\\ & =\int_\Omega h(\det Du(x))\operatorname{cof} Du(x) : D\phi(u(x))Du(x) \, {\rm d}x \\ & =\int_\Omega \operatorname{tr}(D\phi(u(x))) h(\det Du(x))\det Du(x) \, {\rm d}x \\ & =\int_\Omega (\operatorname{div} \phi)(u(x)) |h(\det Du(v(u(x)))| |\det Du(x) |\, {\rm d}x \\ & =\int_{u(\Omega)} \operatorname{div} \phi(y) |h(\det Du(v(y))| N(u|\Omega;y)\,{\rm d}y\\ & =\int_{u(\Omega)}\operatorname{div} \phi(y) |h(\det Du(v(y))| \,{\rm d}y. \end{align*}

This holding for all $\phi \in C^1_c(u(\Omega );\mathbb {R}^n)$ proves that $|h(\det Du(v(y))|$ is constant a.e. on $u(\Omega );$ hence $|h(\det Du)|=\lambda$ is a constant a.e. on $\Omega.$ If $\lambda =0$ then $h(\det Du)=0$ a.e. on $\Omega.$ We now assume $\lambda >0.$ Let

\[ \Omega_+{=}\{x\in\Omega: h(\det Du(x))=\lambda\},\quad \Omega_-{=}\{x\in\Omega: h(\det Du(x))={-}\lambda\}. \]

Then $|\Omega _+|+|\Omega _-|=|\Omega |.$ Since $\lambda >0$ and $h$ is nondecreasing, we have $\det Du\ge 0$ a.e. on $\Omega _+$ and $\det Du\le 0$ a.e. on $\Omega _-.$ We claim that either $|\Omega _+|=0$ or $|\Omega _+|=|\Omega |,$ which proves the theorem. To prove the claim, we observe that, for all $\zeta \in C^1_c(\Omega ;\mathbb {R}^n),$

\[ 0=\int_\Omega h(\det Du)\operatorname{cof} Du : D\zeta=\lambda \int_{\Omega_+} \operatorname{cof} Du : D\zeta -\lambda \int_{\Omega_-} \operatorname{cof} Du : D\zeta; \]

moreover,

\[ 0=\int_\Omega \operatorname{cof} Du : D\zeta = \int_{\Omega_+} \operatorname{cof} Du : D\zeta + \int_{\Omega_-} \operatorname{cof} Du : D\zeta, \]

and hence,

\[ \int_{\Omega_+} \operatorname{cof} Du : D\zeta \,{\rm d}x =0 \quad \forall\ \zeta\in C^1_c(\Omega;\mathbb{R}^n). \]

As above, we take $\zeta (x)=\phi (u(x))$ with arbitrary $\phi \in C^1_c(u(\Omega );\mathbb {R}^n)$ to obtain

\[ 0=\int_{\Omega_+} \operatorname{cof} Du : D\zeta \,{\rm d}x =\int_{u(\Omega_+)} \operatorname{div} \phi(y) \,{\rm d}y =\int_{u(\Omega)} \chi_{u(\Omega_+)}(y)\operatorname{div} \phi(y) \,{\rm d}y. \]

Again, this holding for all $\phi \in C^1_c(u(\Omega );\mathbb {R}^n)$ proves that $\chi _{u(\Omega _+)}$ is constant a.e. on $u(\Omega );$ hence, either $|u(\Omega _+)|=0$ or $|u(\Omega _+)|=|u(\Omega )|.$ If $|u(\Omega _+)|=0,$ then $\int _{\Omega _+} \det Du \,{\rm d}x= |u(\Omega _+)| =0;$ thus $\det Du=0$ a.e. on $\Omega _+,$ which proves $|\Omega _+|=0$ because, otherwise, we would have $\lambda =h(0)=0.$ Similarly, if $|u(\Omega _+)|=|u(\Omega )|$ then $|\Omega _+|=|\Omega |.$ This completes the proof.

Remark 2.9 A sufficient condition for the invertibility of Sobolev functions has been given in [Reference Ball1]. For example, suppose that $\Omega$ is a bounded Lipschitz domain and $u_0\in C(\bar \Omega ;\mathbb {R}^n)$ is such that $u_0$ is one-to-one on $\bar \Omega$ and $u_0(\Omega )$ satisfies the cone condition. Then, condition (2.6) is satisfied provided that

\[ \begin{cases} u|_{\partial\Omega}=u_0,\\ \det Du >0 \text{ a.e.}\,\text{on} \ \Omega,\\ \int_\Omega |\operatorname{cof} Du|^q(\det Du)^{1-q} \,{\rm d}x<\infty\ \mbox{for some}\ q>n. \end{cases} \]

See [Reference Krömer9] for recent studies and more references in this direction.

Proposition 2.10 Let $h(0)=0$, $p\ge n$, $\tilde p={p}/({p-n}),$ and $u\in W^{1,p}(\Omega ;\mathbb {R}^n)$ with $h(\det Du)\in L^{\tilde p}(\Omega )$ be a weak solution of (2.1) satisfying the Dirichlet boundary condition $u|_{\partial \Omega }=Ax,$ where $A\in \mathbb {M}^{n\times n}$ is given. Let

\[ \lambda=\int_\Omega h(\det Du)\det Du\,{\rm d}x, \quad B=\int_\Omega h(\det Du) \operatorname{cof} Du\,{\rm d}x. \]

Then $BA^T=\lambda I.$ Moreover, if $\det A=0$ and $h$ is one-to-one, then $\det Du=0$ a.e. on $\Omega,$ and thus $B=0.$

Proof. For $P\in \mathbb {M}^{n\times n},$ the function $\zeta (x)=P(u(x)-Ax)\in W^{1,p}_0(\Omega ;\mathbb {R}^n)$ is a legitimate test function for (2.2), which, from $D\zeta =PDu-PA$ and $\operatorname {cof} Du: PDu =( \operatorname {tr} P )\det Du,$ yields that

\[ (\operatorname{tr} P) \int_\Omega h(\det Du)\det Du \,{\rm d}x= \int_\Omega h(\det Du)\operatorname{cof} Du:PA\,{\rm d}x. \]

This is simply $(\lambda I-BA^T):P=0.$ Since $P\in \mathbb {M}^{n\times n}$ is arbitrary, we have $BA^T=\lambda I.$ If $\det A=0,$ then $\lambda =0.$ Furthermore, if $h$ is one-to-one, then $\lambda =0$ implies $\det Du=0$ a.e. on $\Omega$ and thus $B=0.$

Remark 2.11

1. (i) Assume $h(0)=0$ and $h$ is one-to-one. If $\det A\ne 0$, then $B=\mu h(\det A)\operatorname {cof} A,$ where $\mu ={\lambda }/{h(\det A)\det A}>0.$ It remains open whether $\mu =1.$ Note that if $\mu \ne 1$ then $\det Du$ cannot be a constant a.e. on $\Omega.$

2. (ii) There are many (very) weak solutions $u\in W^{1,p}(\Omega ;\mathbb {R}^n)$ of equation (2.1) satisfying $u|_{\partial \Omega }=Ax$ for some $p< n$ such that $\det Du$ is not constant a.e. on $\Omega.$ See example 3.4.

Lemma 3.1 Let $p\ge 1$ and $v\in L^p_{loc}(B\setminus \{0\}).$ Define $\tilde v=M(v)\colon (0,\,1)\to \mathbb {R}$ by setting

(3.3)\begin{equation} \tilde v(r) =M(v)(r)= {\int \hspace{-11pt} - \hspace{-1pt}}_{S_r}v(x) \,{\rm d}\sigma_r=\frac{1}{\omega_n} \int_{S_1}v(r\omega) \,{\rm d}\sigma_1, \end{equation}

where $S_r=\partial B_r(0)$, ${\rm d}\sigma _r= {\rm d} \mathcal {H}^{n-1}$ denotes the $(n-1)$-Hausdorff measure on $S_r,$ and $\omega _n=\mathcal {H}^{n-1}(S_1).$ Then $\tilde v\in L^p_{loc}(0,\,1).$ Furthermore, if $v\in W^{1,p}_{loc}(B\setminus \{0\}),$ then $\tilde v\in W^{1,p}_{loc}(0,\,1)$ with

\[ \tilde v'(r)= {1}/{\omega_n} \int_{S_1}Dv(r\omega)\cdot \omega \,{\rm d}\sigma_1 =r^{{-}1}M(Dv\cdot x)(r) \quad a.e.\; r\in (0,1). \]

Since $W^{1,p}_{loc}(0,\,1)\subset C(0,\,1)$, $\tilde v$ can be identified as a continuous function in $(0,\,1)$ if $v\in W^{1,p}_{loc}(B\setminus \{0\}).$

Proof. Note that

\[ |\tilde v(r)| \le {\int \hspace{-11pt} - \hspace{-1pt}}_{S_r}|v| \,{\rm d}\sigma_r \le \left ( {\int \hspace{-11pt} - \hspace{-1pt}}_{S_r}|v|^p \,{\rm d}\sigma_r\right)^{1/p}. \]

Thus, for all $0< a< b<1$,

\[ \int_a^b \omega_nr^{n-1}|\tilde v|^p\,{\rm d}r\le \int_a^b \left ( \int_{S_r}|v|^p \,{\rm d}\sigma_r\right){\rm d}r=\int_{a<|x|< b} |v(x)|^p\,{\rm d}x<\infty. \]

This proves $\tilde v\in L^p_{loc}(0,\,1).$ Now assume $v\in W^{1,p}_{loc}(B\setminus \{0\})$ and let

\[ g(r)= {1}/{\omega_n} \int_{S_1}Dv(r\omega)\cdot \omega \,{\rm d}\sigma_1 =r^{{-}1}M(Dv\cdot x)(r) \in L^p_{loc}(0,1). \]

Let $0< a< b<1$ and $\eta \in C^\infty _c(a,\,b).$ Then

\begin{align*} \int_a^b \tilde v(r)\eta'(r)\,{\rm d}r& =\frac{1}{\omega_n} \int_a^b r^{1-n}\eta'(r) \Big( \int_{S_r} v \,{\rm d}\sigma_r\Big) {\rm d}r\\ & =\frac{1}{\omega_n} \int_{a<|x|< b} |x|^{1-n}\eta'(|x|)v(x)\,{\rm d}x\\ & =\frac{1}{\omega_n} \int_{a<|x|< b} D(\eta(|x|))\cdot (v|x|^{{-}n}x)\,{\rm d}x\\ & ={-}\frac{1}{\omega_n} \int_{a<|x|< b} \eta(|x|) \operatorname{div}(v|x|^{{-}n}x)\,{\rm d}x\\ & ={-}\frac{1}{\omega_n} \int_{a<|x|< b} \eta(|x|) Dv\cdot (|x|^{{-}n}x)\,{\rm d}x\\ & ={-}\frac{1}{\omega_n} \int_a^b \eta(r) r^{{-}n} \Big (\int_{S_r} Dv(x)\cdot x{\rm d}\sigma_r\Big )\,{\rm d}r ={-}\int_a^b g(r)\eta(r)\,{\rm d}r. \end{align*}

This proves $g=\tilde v'.$

Proposition 3.2 Assume $\phi \in W_{loc}^{1,1}(0,\,1)$ and $u(x)=\phi (|x|)x.$ Let $p\ge 1$ and $p'={p}/({p-1}).$ Suppose that $q(r)$ is a measurable function on $(0,\,1)$ such that

(3.4)\begin{equation} q(|x|)\operatorname{cof} Du\in L^p_{loc}(B\setminus \{0\}). \end{equation}

Then, for all $0< a< b<1$ and $v\in W_{loc}^{1,p'}(B;\mathbb {R}^n),$ it follows that

(3.5)\begin{equation} \int_{a<|x|< b} q(|x|)\operatorname{cof} Du:Dv \,{\rm d}x = \omega_n \int_a^b q(r)(r^{n-2}\phi^{n-1}\psi)' \,{\rm d}r, \end{equation}

where $\psi (r)=M(v\cdot x)(r)={r}/{\omega _n} \int _{S_1}v(r\omega )\cdot \omega \,{\rm d}\sigma _1.$

Proof. Note that $q(|x|)\operatorname {cof} Du:Dv\in L^1_{loc}(B\setminus \{0\}).$ Since $\operatorname {cof} Du: Dv =\alpha (r)\operatorname {div} v + \beta (r) Dv:(\omega \otimes \omega ),$ we have

(3.6)\begin{align} \int_{a<|x|< b} q(|x|)\operatorname{cof} Du:Dv \,{\rm d}x & =\int_a^b \Big( \int_{S_r} q(|x|)\operatorname{cof} Du:Dv \,{\rm d}\sigma_r\Big ){\rm d}r \nonumber\\ & = \int_a^b q(r) \Big (\alpha(r) \int_{S_r} \operatorname{div} v\,d \sigma_r\nonumber\\ & \quad + \beta(r) \int_{S_r}Dv:(\omega\otimes \omega) \,{\rm d}\sigma_r \Big ) {\rm d}r. \end{align}

We now compute the two spherical integrals. First, for all $a< t<1,$ by the divergence theorem,

\begin{align*} \int_a^t \int_{S_r} \operatorname{div} v\,{\rm d}\sigma_r \,{\rm d}r & =\int_{a<|x|< t}\operatorname{div} v\,{\rm d}x\\ & =\int_{S_t} v\cdot \frac{x}{t}\,{\rm d}\sigma_t-\int_{S_a} v\cdot \frac{x}{a}\,{\rm d}\sigma_a =\omega_n t^{n-2}\psi(t)-\omega_n a^{n-2}\psi(a). \end{align*}

Hence

\[ \int_{S_r} \operatorname{div} v\,{\rm d}\sigma_r = \omega_n (r^{n-2}\psi(r))' \quad a.e. \ r\in (0,1). \]

Second, since $D(v\cdot x) \cdot x =Dv : (x\otimes x) +v\cdot x,$ we have

\[ \psi'(r)= r^{{-}1} M(D(v\cdot x) \cdot x)= r^{{-}1} M(Dv : (x\otimes x)) + r^{{-}1} M(v\cdot x). \]

Thus $M(Dv : (x\otimes x)) =r \psi ' - \psi$ and hence

\begin{align*} \int_{S_r}Dv:(\omega\otimes \omega) \,{\rm d}\sigma_r & =\omega_n r^{n-3}M(Dv : (x\otimes x)) =\omega_n r^{n-2} \psi' -\omega_n r^{n-3}\psi\\ & = \omega_n (r^{n-2}\psi)' -(n-1)\omega_n r^{n-3}\psi. \end{align*}

Since $\alpha +\beta =\phi ^{n-1}$ and $\beta =-r\phi ^{n-2}\phi '$, elementary computations lead to

\begin{align*} & \alpha(r) \int_{S_r} \operatorname{div} v\,{\rm d}\sigma_r+ \beta(r) \int_{S_r}Dv:(\omega\otimes \omega) \,{\rm d}\sigma_r\\ & \quad = \omega_n \alpha (r^{n-2}\psi)' + \omega_n \beta [(r^{n-2}\psi)' -(n-1) r^{n-3}\psi]=\omega_n (\phi^{n-1}r^{n-2}\psi)' \end{align*}

for a.e. $r\in (0,\,1).$ Finally, (3.5) follows from (3.6).

Theorem 3.3 Assume $h\in C(\mathbb {R})$ is one-to-one, $p\ge {n}/({n-1})$ and $\phi \in W^{1,p}_{loc}(0,\,1).$ Let $u(x)=\phi (|x|)x$ be a weak solution of (2.1) such that

(3.7)\begin{equation} h(\det Du)\operatorname{cof} Du\in L_{loc}^{{p}/{(p-1)}}(B;\mathbb{M}^{n\times n}). \end{equation}

Then either $\phi \equiv 0,$ or

\[ \phi(r)=\left(\lambda + \frac{c}{r^n}\right)^{1/n}\ne 0\quad\forall\ 0< r<1, \]

where $\lambda$ and $c$ are constants. (When $n$ is even, we need $\lambda + {c}/({r^n})> 0$ in $(0,\,1)$ and there are two nonzero branches of the $n$th roots.)

Proof. Let $S=\{r\in (0,\,1):\phi (r)\ne 0\};$ then $S$ is open. If $S=\emptyset,$ then $\phi \equiv 0.$ Assume $S$ is nonempty. Let $(a,\,b)$ be a component of $S.$ Let $\eta (r)\in C_c^\infty (0,\,1)$ be any function with compact support contained in $(a,\,b).$ Define the radial function

\[ \zeta(x)=\begin{cases} \dfrac{\eta(r) }{r^{n}\phi(r)^{n-1}} \,x & \mbox{if} \ r=|x|\in (a,b),\\ 0 & \mbox{otherwise.}\end{cases} \]

Then $\zeta \in W^{1,p}(B;\mathbb {R}^n)$ with $\operatorname {supp} \zeta \subset \{a<|x|< b\}.$ Let $\psi =M(\zeta \cdot x);$ then $r^{n-2}\phi ^{n-1}\psi =\eta.$ Let $\det Du=\phi ^n +r\phi ' \phi ^{n-1}=:d(r).$ By assumption (3.7), $\zeta$ is a legitimate test function for equation (2.1); thus, by (3.5), we obtain that

\begin{align*} 0& = \int_B h(\det Du)\operatorname{cof} Du: D\zeta\,{\rm d}x\\ & = \int_a^b h(d(r)) (r^{n-2}\phi^{n-1}\psi)'\, {\rm d}r =\int_a^b h(d(r))\eta'(r){\rm d}r. \end{align*}

This holds for all $\eta \in C^\infty _c(a,\,b);$ thus $h(d(r))$ is constant a.e. in $(a,\,b)$. As $h$ is one-to-one, we have that $d(r)$ is constant a.e. in $(a,\,b).$ Assume $d(r) =\phi ^n +r\phi ' \phi ^{n-1}= \lambda$ in $(a,\,b).$ Solving the differential equation we obtain that

\[ \phi(r)\ne 0,\quad \phi(r)=\left(\lambda + \frac{c}{r^n}\right)^{1/n} \quad (a< r< b). \]

If one of $a$ and $b$ is inside $(0,\,1),$ then $\phi =0$ at this point; but in this case, $\phi \notin W^{1,q}_{loc}(0,\,1)$ for any $q\ge {n}/{n-1}.$ So $(a,\,b)=(0,\,1);$ this completes the proof.

Proof. If $\lambda _1=0$, then only $Du$ blows up at $r=|x|=b,$ with $|Du(x)|\approx |\phi '(r)|\approx |r-b|^{{1}/({n})-1}$ and $|\operatorname {cof} Du(x)|\approx |\phi (r)|^{n-2}|\phi '(r)|\approx |r-b|^{-{1}/{n}}$ near $r=b.$ Hence, in this case, $u \in W^{1,p}(B;\mathbb {R}^n)$ for $1\le p< {n}/({n-1})$ and $\operatorname {cof} Du\in L^q(B;\mathbb {M}^{n\times n})$ for all $1\le q< n.$

If $\lambda _1\ne 0$, then $u$ and $Du$ also blow up at $x=0$ and $Du$ blows up at $r=a$ and $r=b.$ In this case, the similar blow-up estimates show that $u \in W^{1,p}(B;\mathbb {R}^n)$ and $\operatorname {cof} Du\in L^p(B;\mathbb {M}^{n\times n})$ for $1\le p< {n}/({n-1}).$

Hence, in all cases, $u \in W^{1,p}(B;\mathbb {R}^n)$ and $h(\det Du) \operatorname {cof} Du \in L^p(B;\mathbb {M}^{n\times n})$ for all $1\le p< {n}/({n-1}).$ Given any $\zeta \in C^1(\mathbb {R}^n;\mathbb {R}^n),$ let $\psi (t)= M(\zeta \cdot x)(t)={t}/{\omega _n} \int _{S_1}\zeta (t\omega )\cdot \omega \,{\rm d}\sigma _1.$ Then

\[ \lim_{t\to 0^+} (t^{{-}1}\psi(t))= \lim_{t\to 0^+} \frac{1}{\omega_n} \int_{S_1}\zeta (t\omega)\cdot \omega \,{\rm d}\sigma_1=0. \]

By proposition 3.2, we have

\begin{align*} \int_B h(\det Du) \operatorname{cof} Du:D\zeta & =\lim_{t\to 0^+ } \int_{t<|x|<1} h(\det Du) \operatorname{cof} Du:D\zeta \\ & = h(\lambda_1) \lim_{t\to 0^+} \int_{t<|x|< a} \operatorname{cof} Du:D\zeta\\ & \quad + h(\lambda_2) \int_{b<|x|<1} \operatorname{cof} Du:D\zeta \\ & ={-}\omega_n h(\lambda_1) \lim_{t\to 0^+} t^{n-2}\phi(t)^{n-1}\psi(t)\\ & \quad +\omega_n h(\lambda_2) (t^{n-2}\phi(t)^{n-1}\psi(t))|_b^1\\ & =\omega_n h(\lambda_2) \phi(1)^{n-1}\psi(1). \end{align*}

If $\zeta \in C^1_c(B;\mathbb {R}^n)$, then $\psi (1)=0$; this proves that $u$ is a weak solution of (2.1). Moreover, if $\lambda _2=0$ then $\phi (1)=0;$ in this case, we obtain (3.9).

Definition 3.6 A function $\sigma \colon \mathbb {M}^{n\times n}\to \mathbb {M}^{n\times n}$ is said to be quasimonotone at $A\in \mathbb {M}^{n\times n}$ provided that

\[ \int_\Omega \sigma(A+ D\phi(x)):D\phi(x)\,{\rm d}x \ge 0\quad \forall\ \phi\in C_c^1(\Omega;\mathbb{R}^n). \]

(This condition is independent of the domain $\Omega.$)

Theorem 3.7 Let $n\ge 2$ and $h\in C^1(\mathbb {R})$ be such that

(3.10)\begin{equation} \lambda t^{k_1} \le h'(t)\le \Lambda(t^{k_2}+1)\quad \forall\ t\ge 0, \end{equation}

where $\Lambda >\lambda >0$, $k_1$ and $k_2$ are constants such that $0\le k_1\le k_2< k_1+1.$ Then $\sigma (A)=h(\det A)\operatorname {cof} A$ is not quasimonotone at $I\in \mathbb {M}^{n\times n}.$

Proof. Let $\Omega$ be the unit ball in $\mathbb {R}^n.$ We show that there exists a radial function $\phi (x)=\rho (|x|)x,\,$ where $\rho \in W^{1,\infty }(0,\,1)$ with $\rho (1)=0,$ such that

(3.11)\begin{equation} \int_\Omega \sigma(I+ D\phi(x)):D\phi(x)\,{\rm d}x<0. \end{equation}

Using the spherical coordinates, we compute that

\[ \int_\Omega \sigma(I+ D\phi(x)):D\phi(x)\,{\rm d}x =\omega_n \int_0^1 P(r)\,{\rm d}r, \]

where

\begin{align*} P(r)& =h\Big ((1+ \rho)^{n} + (1+ \rho)^{n-1} \rho' r\Big ) \Big ( n\rho (1+\rho)^{n-1} +(1+n\rho)(1+\rho)^{n-2} \rho' r\Big)r^{n-1}\\ & =h(A+B)(C+D), \end{align*}

with $A =(1+\rho )^n,\, B =(1+\rho )^{n-1}\rho ' r,\, C =n\rho (1+\rho )^{n-1}r^{n-1}$ and $D =(1+n\rho) (1+\rho )^{n-2}\rho 'r^n.$ We write $h(A+B)=h(A+B)-h(A)+h(A)=EB+h(A),$ where

\[ E= \int_0^1h'(A +tB )\,{\rm d}t. \]

Thus $P=EBC+EBD+h(A)C+h(A)D.$

Let $0< a<1$ be fixed and $b=a-\epsilon$ with $0<\epsilon < a$ sufficiently small. Define

(3.12)\begin{equation} \rho=\rho_\epsilon(r)=\begin{cases} -1, & 0\le r\le b,\\ \frac{n-1}{n\epsilon }(r-a)-\frac{1}{n}, & b\le r \le a,\\ \frac{1}{n(1-a)} (r-1), & a\le r\le 1. \end{cases} \end{equation}

(See Fig. 1.) Then, with $P(r)=P_\epsilon (r)$, we have

(3.13)\begin{equation} \int_0^1 P_\epsilon (r)\,{\rm d}r=\int_b^a P(r)\,{\rm d}r+\int_a^1 P(r)\,{\rm d}r,\quad \Big |\int_a^1 P(r)\,{\rm d}r \Big |\le M_1, \end{equation}

where $M_1$ (likewise, each of the $M_k$'s below) is a positive constant independent of $\epsilon.$ For all $b< r< a$ we have $\rho '=({n-1})/{n\epsilon }$, $n\rho +1=({n-1})/{ \epsilon } (r-a)$ and $\rho +1 =({n-1}/{n\epsilon } (r-b).$ Hence $0< A<1$ and $B> 0$ on $(b,\,a)$; moreover,

(3.14)\begin{equation} \Big |\int_b^a \Big ( h(A)C+h(A)D \Big )\,{\rm d}r \Big |\le M_2. \end{equation}

Figure 1. The graph of $\rho =\rho _\epsilon (r),$ where $0< a<1$ is fixed and $b=a-\epsilon$ with $\epsilon \in (0,\,a)$ sufficiently small.

Moreover,

\[ \lambda\int_0^1 (A+tB)^{k_1} \,{\rm d}t \le E\le \Lambda \left (1+\int_0^1(A+tB)^{k_2} \,{\rm d}t\right ). \]

Since

\[ \int_0^1 (A+t B)^k \,{\rm d}t = \epsilon^{{-}k} \int_0^1 \big (\epsilon A+ (1+\rho)^{n-1}\frac{n-1}{n} r t \big )^k \,{\rm d}t, \]

it follows that for all $k\ge 0$

\[ M_3 \epsilon^{{-}k} (1+\rho)^{k(n-1)}r^k \le \int_0^1 (A+t B)^k \,{\rm d}t \le M_4 \epsilon^{{-}k}. \]

Thus

(3.15)\begin{equation} \Big |\int_b^a EBC \,{\rm d}r \Big |\le M_5 (1+ \epsilon^{{-}k_2}). \end{equation}

Since $BD\le 0$ on $(b,\,a)$, we have

\begin{align*} & \int_b^a EBD\,{\rm d}r \le \lambda \int_b^a M_3 \epsilon^{{-}k_1} (1+\rho)^{k_1(n-1)}r^{k_1} BD\,{\rm d}r\\ & \quad =\lambda M_3 \epsilon^{{-}k_1} \int_b^a(1+\rho)^{k_1(n-1)+2n-3}(n\rho +1)\rho'^2 r^{k_1+n+1}\,{\rm d}r\\ & \quad \le \frac{ b^{k_1+n+1}M_6}{\epsilon^{2n+k_1n}} \int_b^a (r-b)^{k_1(n-1)+2n-3}(r-a) \,{\rm d}r ={-}M_7\epsilon^{{-}k_1-1}. \end{align*}

This, combined with (3.13)–(3.15), proves that

\[ \lim_{\epsilon\to 0^+} \left (\epsilon^{k_2} \int_0^1 P_\epsilon (r)\,{\rm d}r\right ) \le \lim_{\epsilon\to 0^+} (M_5 -M_7 \epsilon^{k_2-k_1-1}) ={-}\infty. \]

Consequently, $\int _0^1 P_\epsilon (r)\,{\rm d}r<0$ if $\epsilon \in (0,\,a)$ is sufficiently small; this establishes (3.11).

Proposition 4.3 Assume $h$ is one-to-one. Let $\begin{bmatrix}A\\ {B}\end{bmatrix}\in \mathcal {K}^{qc}.$ Then ${B=\mu h(\det A) A}$ for some $\mu >0;$ in particular, $B=0$ if $\det A=0.$

Proof. Let $\{u_n\}$ and $\{v_n\}$ be uniformly bounded in $W^{1,\infty }(\Omega ;\mathbb {R}^2)$ such that

(4.5)\begin{equation} u_n|_{\partial\Omega}=Ax, \quad v_n|_{\partial\Omega}=Bx, \quad \lim_{n\to \infty}\int_\Omega \operatorname{dist} \left (\begin{bmatrix} Du_n \\ Dv_n\end{bmatrix}, \mathcal{K} \right) \,{\rm d}x =0. \end{equation}

In what follows, for any two vectors $\chi ^1$ and $\chi ^2$ in $\mathbb {R}^2$, we define $\chi ^1\wedge \chi ^2=\det X,$ where $X$ is the matrix in $\mathbb {M}^{2\times 2}$ having $\chi ^1$ and $\chi ^2$ as its first and second rows. For $\eta \in \mathbb {M}^{4\times 2},$ let $\eta ^i\in \mathbb {R}^2$ be its $i$th row vector for $1\le i\le 4.$ Consider functions: $H_1(\eta )=|h(\eta ^1\wedge \eta ^2) (\eta ^1\wedge \eta ^2) -(\eta ^1\wedge \eta ^4)|$, $H_2(\eta )=|h(\eta ^1\wedge \eta ^2) (\eta ^1\wedge \eta ^2) -(\eta ^3\wedge \eta ^2)|$, and $H_3(\eta )=|\eta ^1\wedge \eta ^3|+| \eta ^2\wedge \eta ^4|;$ they are all nonnegative continuous and vanish on the set $\mathcal {K}.$ By (4.5) and remark 4.2(i), we have

(4.6)\begin{align} & \lim_{n\to \infty}\int_\Omega |h(\det Du_n) Du_n - Dv_n | \,{\rm d}x=0,\nonumber\\ & \lim_{n\to \infty}\int_\Omega |h(\alpha_n^1 \wedge \alpha_n^2)(\alpha_n^1\wedge \alpha_n^2) -\alpha_n^1\wedge \beta_n^2| \,{\rm d}x=0, \nonumber\\ & \lim_{n\to \infty}\int_\Omega |h(\alpha_n^1 \wedge \alpha_n^2)(\alpha_n^1\wedge \alpha_n^2) -\beta_n^1\wedge \alpha_n^2| \,{\rm d}x=0,\nonumber\\ & \lim_{n\to \infty}\int_\Omega (|\alpha_n^1\wedge \beta_n^1|+ |\alpha_n^2\wedge \beta_n^2|) \,{\rm d}x=0, \end{align}

where $\alpha _n^i$ and $\beta _n^i$ are the $i$th row of $Du_n$ and $Dv_n,$ respectively, for $i=1,\,2.$ Let $\alpha ^i$ and $\beta ^i$ be the $i$th row of $A$ and $B,$ respectively, for $i=1,\,2.$ Then, by the boundary conditions in (4.5) and the null-Lagrangian property of $2\times 2$ minors, we have $\int _\Omega \alpha _n^i\wedge \beta _n^k \,{\rm d}x = (\alpha ^i\wedge \beta ^k) |\Omega |$ for $i,\,k=1,\,2.$ Thus, it follows from (4.6) that

(4.7)\begin{equation} \alpha^1\wedge \beta^1 = \alpha^2\wedge \beta^2 =0, \quad \beta^1\wedge \alpha^2= \alpha ^1\wedge \beta ^2 = \lambda, \end{equation}

where

\[ \lambda = \lim_{n\to \infty} {\int \hspace{-11pt} - \hspace{-1pt}}_\Omega h(\alpha_n^1\wedge \alpha_n^2) (\alpha_n^1\wedge \alpha_n^2) \,{\rm d}x. \]

Case 1: $\lambda =0.$ Since $h(0)=0$ and $h$ is one-to-one, we have that either $h(t)t >0$ for all $t\ne 0$ or $h(t)t <0$ for all $t\ne 0.$ In this case,

\[ \lim_{n\to \infty} {\int \hspace{-11pt} - \hspace{-1pt}}_\Omega |h(\alpha_n^1\wedge \alpha_n^2)(\alpha_n^1\wedge \alpha_n^2)|\,{\rm d}x =|\lambda|=0. \]

Via a subsequence of $n\to \infty,$ we have $h(\alpha _n^1\wedge \alpha _n^2)(\alpha _n^1\wedge \alpha _n^2)\to 0;$ thus $\alpha _n^1\wedge \alpha _n^2=\det Du_n \to 0$ a.e. on $\Omega.$ Hence $\det A={1}/{|\Omega |}\int _\Omega \det Du_n \to 0$ and $\det A=0.$ Moreover, since $h(\det Du_n)\to 0$ a.e. on $\Omega,$ we have

\[ |B| = \lim_{n\to \infty} \Big | {\int \hspace{-11pt} - \hspace{-1pt}}_\Omega Dv_n \,{\rm d}x \Big | \le \lim_{n\to \infty} {\int \hspace{-11pt} - \hspace{-1pt}}_\Omega |h(\det Du_n) Du_n - Dv_n | \,{\rm d}x =0. \]

So, in this case, we have $\det A=0$, $B=0,$ and thus $B=\mu h(\det A) A$ for all $\mu >0.$

Case 2: $\lambda \ne 0.$ In this case, by (4.7), both $\alpha ^1$ and $\alpha ^2$ are nonzero, and we have $\beta ^1=t\alpha ^1$ and $\beta ^2=s\alpha ^2$ for some constants $t,\,s\in \mathbb {R};$ thus $\alpha ^1\wedge \beta ^2=\beta ^1\wedge \alpha ^2=t\alpha ^1\wedge \alpha ^2=s\alpha ^1\wedge \alpha ^2= \lambda \ne 0.$ It follows that $\det A=\alpha ^1\wedge \alpha ^2\ne 0$ and $t=s= {\lambda }/{\det A}.$ Therefore, in this case, we have $B=tA=\mu h(\det A) A,$ where $\mu = {\lambda }/{h(\det A)\det A}>0,\,$ because the signs of $h(t)t$ and $\lambda$ are the same for all $t\ne 0.$

Finally, from the two cases above, we see that $\det A=0$ if and only if $\lambda =0.$ Thus $B=0$ if $\det A=0.$

Remark 4.4 Let $\Omega$ be the unit open disc in $\mathbb {R}^2.$ Given any $\mu >1,$ let $\lambda >1$ be the unique number such that $h(\lambda )=\mu h(1).$ Define the radial functions $u(x)= \phi (|x|) x$ and $v(x)= h(\lambda ) u (x),$ where

\[ \phi(r)=\begin{cases} 0 & \mbox{if} \ 0\le r\le \sqrt{\frac{\lambda-1}{\lambda }},\\ \sqrt{\lambda -\frac{\lambda -1}{r^2}} & \mbox{if} \ \sqrt{\frac{\lambda -1}{\lambda }}\le r\le 1. \end{cases} \]

It follows from example 3.4 that $(u,\,v)\in C(\bar \Omega ;\mathbb {R}^4)\cap W^{1,p}(\Omega ;\mathbb {R}^4)$ for all $1\le p<2$ and is a solution of (4.9) with $A=I,$ but $(u,\,v)\notin W^{1,p}(\Omega ;\mathbb {R}^4)$ for any $p\ge 2.$

However, it remains open whether problem (4.9) has a Lipschitz or even a $W^{1,2}(\Omega ;\mathbb {R}^4)$ solution $(u,\,v)$ for some positive $\mu \ne 1.$

Theorem 4.5 Assume $h$ is one-to-one and $\det A \ne 0.$ Let $\mu >0$ and suppose that $(u,\,v)\colon \bar \Omega \to \mathbb {R}^4$ is a Lipschitz solution to problem (4.9) such that $(\det A) {\det Du} \ge 0$ a.e. in $\Omega.$ Then, $\det Du=\det A$ a.e. on $\Omega$, $\mu =1$, and $v=h(\det A) u$ on $\bar \Omega.$

Proof. Write $g=h(\det Du).$ Since $gDu=Dv$, we have

\[ g^2\det Du=\det Dv,\quad 2g\det Du= Dv:\operatorname{cof} Du \quad a.e.\ \Omega. \]

Integrating over $\Omega$ and using the null-Lagrangian property of $2\times 2$ minors, we have

\begin{align*} {\int \hspace{-11pt} - \hspace{-1pt}}_\Omega \det Du & =\det A,\quad {\int \hspace{-11pt} - \hspace{-1pt}}_\Omega g \det Du =\mu h(\det A)\det A,\\ {\int \hspace{-11pt} - \hspace{-1pt}}_\Omega g^2 \det Du & =(\mu h(\det A))^2 \det A, \end{align*}

and thus

\begin{align*} & \int_\Omega (g-\mu h(\det A))^2(\det A) {\det Du} \,{\rm d}x\\ & \quad =(\det A) \int_\Omega \Big ( g^2 \det Du -2\mu h(\det A) g\det Du +(\mu h(\det A))^2 \det Du\Big )\,{\rm d}x =0. \end{align*}

Since $(\det A) {\det Du} \ge 0$ a.e. in $\Omega,$ it follows that $g =h(\det Du)=\mu h(\det A)$ a.e. on the set $E=\{x\in \Omega : (\det A) \det Du(x)> 0\}.$ As $h$ is one-to-one, we have $\det Du=\lambda \chi _E$, where $\lambda \ne 0$ is the unique number such that $h(\lambda )=\mu h(\det A).$ Thus $Dv = h(\lambda ) (Du) \chi _E$ and $\det Dv=(h(\lambda ))^2\lambda \chi _E.$ Let $\tilde v (x)=Pv(x),$ where $P=\mbox {diag}(1,\,{1}/{(h(\lambda ))^2\lambda })$ is a diagonal matrix. Then $\tilde v$ is Lipschitz on $\Omega$, $D\tilde v = h(\lambda ) P(Du) \chi _E$, and $\det D\tilde v= \chi _E;$ thus,

\[ |D\tilde v(x)|^2\le L\det D\tilde v(x) \quad \mbox{ a.e.}\ \text{on} \ \Omega, \]

where $L= \|D\tilde v\|^2_{L^\infty (\Omega )}$. Since $\det Du=\lambda \chi _E$, we have $\lambda |E| = \int _\Omega \det Du =|\Omega |\det A \ne 0;$ thus $|E|>0.$ Hence $\tilde v$ is not constant on $\Omega.$ This proves that $\tilde v$ is a non-constant $L$-quasiregular map on $\Omega.$ It is well-known (see [Reference Lehto and Virtanen12, Reference Reshetnyak15]) that any non-constant $L$-quasiregular mapping cannot have its Jacobian determinant equal to zero on a set of positive measure; hence $|\Omega \setminus E|=0,$ which implies that $\det Du=\lambda,$ $h(\lambda ) Du=Dv$ a.e. on $\Omega,$ and $\lambda ={1}/{|\Omega |}\int _\Omega \det Du =\det A.$ Thus, from $h(\lambda )=\mu h(\det A),$ we have $\mu = 1.$ Finally, since $Dv=h(\lambda ) Du$ a.e. on $\Omega$ and $v= h(\lambda )u$ on $\partial \Omega,$ it follows that $v=h(\lambda )u$ on $\bar \Omega.$

Theorem 4.6 Let $h$ be one-to-one. Then $\mathcal {K}_\epsilon ^{qc}=\mathcal {K}_\epsilon$ for all $\epsilon >0.$

Proof. Let $\begin{bmatrix} A\\ {B}\end{bmatrix} \in \mathcal {K}_\epsilon ^{qc}.$ Then there exist sequences $\{u_n\}$ and $\{v_n\}$ uniformly bounded in $W^{1,\infty }(\Omega ;\mathbb {R}^2)$ such that

(4.11)\begin{equation} u_n|_{\partial\Omega}=Ax, \quad v_n|_{\partial\Omega}=Bx, \quad \lim_{n\to \infty}\int_\Omega \operatorname{dist} \left(\begin{bmatrix} Du_n \\ Dv_n\end{bmatrix}, \mathcal{K}_\epsilon\right) \,{\rm d}x =0. \end{equation}

Since $\mathcal {K}_\epsilon$ is closed, it follows that there exist measurable functions $A_n\colon \Omega \to \mathbb {M}^{2\times 2}$ with $\det A_n(x)\ge \epsilon$ a.e. $x\in \Omega$ such that

\[ \operatorname{dist} \left(\begin{bmatrix} Du_n \\ Dv_n\end{bmatrix}, \mathcal{K}_\epsilon\right) =\left[|Du_n-A_n|^2 +|Dv_n-h(\det A_n)A_n|^2\right]^{1/2}. \]

It is easily seen that $\{A_n\}$ is uniformly bounded in $L^\infty (\Omega ; \mathbb {M}^{2\times 2}).$ Thus, we have

(4.12)\begin{align} & \det A=\lim_{n\to \infty} {\int \hspace{-11pt} - \hspace{-1pt}}_\Omega \det Du_n\,{\rm d}x =\lim_{n\to \infty} {\int \hspace{-11pt} - \hspace{-1pt}}_\Omega \det A_n\,{\rm d}x \ge \epsilon,\nonumber\\ & \lim_{n\to \infty}\int_\Omega |h(\det Du_n) Du_n - Dv_n | \,{\rm d}x=0. \end{align}

By proposition 4.3, we have $B=tA$ for some number $t.$ Note that ${1}/{|\Omega |}\int _\Omega \det Du_n =\det A$ and, as in the proof of proposition 4.3,

\[ \lim_{n\to \infty} {\int \hspace{-11pt} - \hspace{-1pt}}_\Omega h(\det Du_n) \det Du_n \,{\rm d}x = t\det A; \]

furthermore,

\[ \lim_{n\to \infty} {\int \hspace{-11pt} - \hspace{-1pt}}_\Omega (h(\det Du_n))^2\det Du_n \,{\rm d}x = \lim_{n\to \infty} {\int \hspace{-11pt} - \hspace{-1pt}}_\Omega \det Dv_n \,{\rm d}x =t^2 \det A. \]

Hence

\[ \lim_{n\to \infty} {\int \hspace{-11pt} - \hspace{-1pt}}_\Omega (h(\det Du_n) -t)^2\det A_n \,{\rm d}x = \lim_{n\to \infty} {\int \hspace{-11pt} - \hspace{-1pt}}_\Omega (h(\det Du_n) -t)^2\det Du_n \,{\rm d}x =0. \]

Since $\det A_n\ge \epsilon,$ we obtain

\[ \lim_{n\to \infty} {\int \hspace{-11pt} - \hspace{-1pt}}_\Omega (h(\det Du_n) -t)^2 \,{\rm d}x =0, \]

Via a subsequence $n_j\to \infty$, we have $h(\det Du_{n_j}(x))\to t$ for a.e. $x\in \Omega.$ Since $h$ is continuous and one-to-one, this implies $\det Du_{n_j}(x)\to h^{-1}(t)$ for a.e. $x\in \Omega.$ Thus,

\[ \det A= {\int \hspace{-11pt} - \hspace{-1pt}}_\Omega \det Du_{n_j}(x)\,{\rm d}x \to h^{{-}1}(t) \]

and $t=h(\det A).$ This proves $B=h(\det A)A;$ thus $\begin{bmatrix} A\\ {B}\end{bmatrix} \in \mathcal {K}_\epsilon,$ as $\det A\ge \epsilon.$

Proposition 4.7 Let $h$ be one-to-one. Suppose that sequences $\{u_n\}$ and $\{v_n\}$ converge weakly* to $\bar u$ and $\bar v$ in $W^{1,\infty }(\Omega ;\mathbb {R}^2)$ as $n\to \infty,$ respectively, and satisfy

(4.13)\begin{equation} \lim_{n\to \infty}\int_\Omega \operatorname{dist} \left(\begin{bmatrix} Du_n \\ Dv_n\end{bmatrix}, \mathcal{K}_\epsilon\right) \,{\rm d}x =0. \end{equation}

Then $\begin{bmatrix} D\bar u (x) \\ D\bar v(x)\end{bmatrix} \in \mathcal {K}_\epsilon$ for a.e. $x\in \Omega ;$ moreover, $\det Du_n \to \det D\bar u$ strongly in $L^p(\Omega )$ for all $1\le p<\infty.$

Proof. Note that the stated inclusion follows from theorem 4.6 by a general theorem on quasiconvex hulls; however, we give a direct proof without using such a result. As in the proof of theorem 4.6, let $\{A_n\}$ be a uniformly bounded sequence in $L^\infty (\Omega ; \mathbb {M}^{2\times 2})$ with $\det A_n(x)\ge \epsilon$ for a.e. $x\in \Omega$ such that

(4.14)\begin{equation} \lim_{n\to \infty}\int_\Omega \left[|Du_n-A_n|^2 +|Dv_n-h(\det A_n)A_n|^2\right]^{1/2} {\rm d}x =0. \end{equation}

Thus, for all measurable sets $E\subset \Omega,$

\[ \int_E\det D\bar u \,{\rm d}x =\lim_{n\to\infty}\int_E \det Du_n\,{\rm d}x =\lim_{n\to\infty}\int_E \det A_n\,{\rm d}x \ge \epsilon |E|. \]

From this it follows that

\[ \det D\bar u(x)\ge \epsilon \quad \mbox{a.e.}\ x\in \Omega. \]

As before, let $\alpha ^i$ and $\beta ^i$ be the $i$th row of $D\bar u$ and $D\bar v,$ respectively, and let $\alpha _n^i$ and $\beta _n^i$ be the $i$th row of $Du_n$ and $Dv_n,$ respectively, for $i=1,\,2.$ Then all the limits listed in (4.6) above still hold. By the weak* convergence of minors, for all measurable sets $E\subset \Omega,$ we have

\[ \Big |\int_E \alpha^1\wedge \beta^1 \Big |=\Big | \lim_{n\to\infty}\int_E \alpha_n^1\wedge \beta_n^1 \Big |\le \lim_{n\to\infty}\int_\Omega |\alpha_n^1\wedge \beta_n^1| =0, \]

and similarly,

\begin{align*} \Big |\int_E \alpha^2\wedge \beta^2 \Big | & \le \lim_{n\to\infty}\int_\Omega |\alpha_n^2\wedge \beta_n^2| =0,\\ \Big |\int_E (\alpha^1\wedge \beta^2-\beta^1\wedge \alpha^2) \Big | & \le \lim_{n\to\infty}\int_\Omega |\alpha_n^1\wedge \beta_n^2-\beta_n^1\wedge \alpha_n^2| =0. \end{align*}

So $\alpha ^1\wedge \beta ^1=\alpha ^2\wedge \beta ^2=0$ and $\alpha ^1\wedge \beta ^2=\beta ^1\wedge \alpha ^2$ a.e. on $\Omega.$ Since $\alpha ^1\wedge \alpha ^2=\det D\bar u \ge \epsilon,$ it follows that $\beta ^1=g\alpha ^1$ and $\beta ^2=g\alpha ^2,$ where $g={\alpha ^1\wedge \beta ^2}/{\alpha ^1\wedge \alpha ^2}\in L^\infty (\Omega ).$ Again by (4.6) and the weak* convergence of $2\times 2$ minors, we have

\begin{align*} \lim_{n\to \infty}\int_\Omega g^2 \det Du_n\,{\rm d}x& =\int_\Omega g^2 \det D\bar u\,{\rm d}x,\\ \lim_{n\to \infty}\int_\Omega g h(\det Du_n) \det Du_n \,{\rm d}x & = \lim_{n\to \infty}\int_\Omega g \beta_n^1\wedge \alpha_n^2 \,{\rm d}x = \int_\Omega g^2\det D\bar u\,{\rm d}x,\\ \lim_{n\to \infty}\int_\Omega (h(\det Du_n))^2\det Du_n \,{\rm d}x & = \lim_{n\to \infty}\int_\Omega \det Dv_n \,{\rm d}x = \int_\Omega g^2\det D\bar u\,{\rm d}x. \end{align*}

Therefore,

\[ \lim_{n\to \infty}\int_\Omega (h(\det Du_n) -g)^2\det A_n \,{\rm d}x = \lim_{n\to \infty}\int_\Omega (h(\det Du_n) -g)^2\det Du_n \,{\rm d}x =0. \]

Since $\det A_n\ge \epsilon$, it follows that $h(\det Du_n) \to g$ strongly in $L^2(\Omega ).$ We assume, along a subsequence $n_j\to \infty,$ that $h(\det Du_{n_j}(x))\to g(x)$ for a.e. $x\in \Omega.$ Since $h$ is continuous and one-to-one, we have $\det Du_{n_j}(x)\to h^{-1}(g(x))$ for a.e. $x\in \Omega.$ But, since $\det Du_n\rightharpoonup \det D\bar u$ weakly* in $L^\infty (\Omega ),$ it follows that $h^{-1}(g(x))=\det D\bar u(x)$ a.e. $x\in \Omega,$ which proves $g=h(\det D\bar u)$ and hence $D\bar v= h(\det D\bar u)D\bar u.$ Finally, the strong convergence of $h(\det Du_n) \to h(\det D\bar u)$ in $L^2(\Omega )$ and the weak* convergence of $\{u_n\}$ and $\{v_n\}$ in $W^{1,\infty }(\Omega ;\mathbb {R}^2)$ establish the strong convergence of $\det Du_n\to \det D\bar u$ in $L^p(\Omega )$ for all $1\le p<\infty.$