2.1 Symmetric functions

Integer partitions are written $\lambda = (\lambda _{1}\geq \cdots \geq \lambda _{l})$ , possibly with trailing zeroes. We set $|\lambda | = \lambda _{1}+\cdots +\lambda _{l}$ and let $\ell (\lambda )$ be the number of nonzero parts. We also define

(6) $$ \begin{align} n(\lambda ) = \sum _{i} (i-1)\lambda _{i}. \end{align} $$

The transpose of $\lambda $ is denoted $\lambda ^{*}$ .

The partitions of a given integer n are partially ordered by

(7) $$ \begin{align} \lambda \leq \mu \quad \text{if}\quad \lambda _{1}+\cdots +\lambda _{k} \leq \mu _{1}+\cdots +\mu _{k} \text{ for all } k, \end{align} $$

where the sums include trailing zeroes for $k>\ell (\lambda )$ or $k>\ell (\mu )$ .

The (French style) Young or Ferrers diagram of a partition $\lambda $ is the set of lattice points $\{(i,j)\mid 1\leq j\leq \ell (\lambda ),\; 1\leq i \leq \lambda _{j} \}$ . We often identify $\lambda $ and its diagram with the set of lattice squares, or boxes, with northeast corner at a point $(i,j)\in \lambda $ . A skew diagram (or skew shape) $\lambda /\mu $ is the difference between the diagram of a partition $\lambda $ and that of a partition $\mu \subseteq \lambda $ contained in it. The diagram generator of $\lambda $ is the polynomial

(8) $$ \begin{align} B_{\lambda }(q,t) = \sum _{(i,j)\in \lambda} q^{i-1}\, t^{j-1}. \end{align} $$

Let $\Lambda = \Lambda _{{\mathbf k} }(X)$ be the algebra of symmetric functions in an infinite alphabet of variables $X = x_{1},x_{2},\ldots $ , with coefficients in the field ${\mathbf k} = {\mathbb Q} (q,t)$ . We follow Macdonald’s notation [Reference Macdonald26] for various graded bases of $\Lambda $ , such as the elementary symmetric functions $e_{\lambda } = e_{\lambda _{1}}\cdots e_{\lambda _{k}}$ , complete homogeneous symmetric functions $h_{\lambda } = h_{\lambda _{1}}\cdots h_{\lambda _{k}}$ , power-sums $p_{\lambda } = p_{\lambda _{1}}\cdots p_{\lambda _{k}}$ , monomial symmetric functions $m_{\lambda }$ and Schur functions $s_{\lambda }$ . The involutory ${\mathbf k} $ -algebra automorphism $\omega \colon \Lambda \rightarrow \Lambda $ mentioned in the introduction may be defined by any of the formulas

(9) $$ \begin{align} \omega \, e_{k} = h_{k},\quad \omega \, h_{k} = e_{k},\quad \omega \, p_{k} = (-1)^{k-1} p_{k},\quad \omega \, s_{\lambda } = s_{\lambda ^{*}}. \end{align} $$

We also need the symmetric bilinear inner product $\langle -, - \rangle $ on $\Lambda $ defined by any of

(10) $$ \begin{align} \langle s_{\lambda },s_{\mu } \rangle = \delta _{\lambda ,\mu },\qquad \langle h_{\lambda },m_{\mu } \rangle = \delta _{\lambda ,\mu },\qquad \langle p_{\lambda },p_{\mu } \rangle = z_{\lambda }\, \delta _{\lambda ,\mu }, \end{align} $$

where $z_{\lambda } = \prod _{i} r_{i}!\, i^{r_{i}}$ if $\lambda = (1^{r_{1}},2^{r_{2}},\ldots )$ .

We write $f^{\bullet }$ for the operator of multiplication by a function f. Otherwise, the custom of writing f for both the operator and the function would make it hard to distinguish between operator expressions such as $(\omega f)^{\bullet }$ and $\omega \cdot f^{\bullet }$ . When f is a symmetric function, we write $f^{\perp }$ for the $\langle -, - \rangle $ adjoint of $f^{\bullet }$ .

2.2 Plethystic evaluation

We briefly recall the device of plethystic evaluation. If A is an expression in terms of indeterminates, such as a polynomial, rational function, or formal series, we define $p_{k}[A]$ to be the result of substituting $a^{k}$ for every indeterminate a occurring in A. We define $f[A]$ for any $f\in \Lambda $ by substituting $p_{k}[A]$ for $p_{k}$ in the expression for f as a polynomial in the power-sums $p_{k}$ , so that $f\mapsto f[A]$ is a homomorphism.

The variables $q, t$ from our ground field ${\mathbf k} $ count as indeterminates.

As a simple example, the plethystic evaluation $f[x_{1}+\cdots +x_{l}]$ is just the ordinary evaluation $f(x_{1},\ldots ,x_{l})$ , since $p_{k}[x_{1}+\cdots +x_{l}] = x_{1}^{k}+\cdots +x_{l}^{k}$ . This also works in infinitely many variables.

When $X = x_{1},x_{2},\ldots $ is the name of an infinite alphabet of variables, we use $f(X)$ , with round brackets, as an abbreviation for $f(x_{1},x_{2},\ldots )\in \Lambda (X)$ . In this situation, we also make the convention that when X appears inside plethystic brackets, it means $X = x_{1}+x_{2}+\cdots $ . With this convention, $f[X]$ is another way of writing $f(X)$ .

As a second example and caution to the reader, the formula in equation (9) for $\omega \, p_{k}$ implies the identity $\omega f(X) = f[-z X]|_{z=-1}$ . Note that $f[-z X]|_{z=-1}$ does not reduce to $f(X)$ , as it might at first appear, since specializing the indeterminate z to a number does not commute with plethystic evaluation.

Plethystic evaluation of a symmetric infinite series is allowed if the result converges as a formal series. The series

(11) $$ \begin{align} \Omega = 1 + \sum _{k>0} h_{k} = \exp \sum _{k>0} \frac{p_{k}}{k}, \quad \text{or}\quad \Omega [a_{1}+a_{2}+\cdots -b_{1}-b_{2}-\cdots ] = \frac{\prod_{i} (1-b_{i})}{\prod_{i} (1-a_{i})} \end{align} $$

is particularly useful. The classical Cauchy identity can be written using this notation as

(12) $$ \begin{align} \Omega [XY] = \sum _{\lambda }s_{\lambda }[X]s_{\lambda }[Y]. \end{align} $$

Taking the inner product with $f(X)$ in equation (12) yields $f[A] = \langle \Omega [AX], f(X) \rangle $ , which implies

(13) $$ \begin{align} \langle \Omega [AX]\Omega [BX],f(X) \rangle = f[A+B] = \langle \Omega [BX],f[X+A] \rangle. \end{align} $$

As B is arbitrary, $\Omega [BX]$ is in effect a general symmetric function, so equation (13) implies

(14) $$ \begin{align} \Omega [AX]^{\perp }f(X) = f[X+A]. \end{align} $$

Note that, although $\Omega [AX]^{\perp } = \sum _{k} h_{k}[AX]^{\perp }$ is an infinite series, it converges formally as an operator applied to any $f\in \Lambda (X)$ , since $h_{k}[AX]^{\perp }$ has degree $-k$ , and so kills f for $k \gg 0$ .

Identifying $\Lambda $ with a polynomial ring in the power-sums $p_{k}$ , we have

(15) $$ \begin{align} p_{k}^{\perp } = k \frac{\partial }{\partial p_{k}}. \end{align} $$

In fact, taking $A = z$ and $f = p_{k}$ in (14) shows that $\exp ( \sum (p_{k}^{\perp } z^{k})/k)$ is the operator that substitutes $p_{k}+z^{k}$ for $p_{k}$ in any polynomial $f(p_{1},p_{2},\ldots )$ . This operator can also be written $\exp (\sum z^{k} \frac {\partial }{\partial p_{k}})$ , giving (15).

Another consequence of equation (14) is the operator identity

(16) $$ \begin{align} \Omega [AX]^{\perp }\, \Omega [BX]^{\bullet } = \Omega [AB]\, \Omega [BX]^{\bullet }\, \Omega [AX]^{\perp } \end{align} $$

with notation $\Omega [BX]^{\bullet }$ as in §2.1.

2.3 $\operatorname {\mathrm {GL}}_{l}$ characters

The weight lattice of $\operatorname {\mathrm {GL}} _{l}$ is $X = {\mathbb Z} ^{l}$ , with Weyl group $W = S_{l}$ permuting the coordinates. Letting $\varepsilon _{1},\ldots ,\varepsilon _{l}$ be unit coordinate vectors, the positive roots are $\varepsilon _{i} - \varepsilon _{j}$ for $i<j$ , with simple roots $\alpha _{i} = \varepsilon _{i} - \varepsilon _{i+1}$ for $i=1,\ldots ,l-1$ . The standard pairing on ${\mathbb Z} ^{l}$ in which the $\varepsilon _{i}$ are orthonormal identifies the dual lattice $X^{*}$ with X. Under this identification, the coroots coincide with the roots, and the simple coroots $\alpha _{i}^{\vee } $ with the simple roots. A weight $\lambda \in {\mathbb Z} ^{l}$ is dominant if $\lambda _{1}\geq \cdots \geq \lambda _{l}$ ; a weight is regular (has trivial stabilizer in $S_{l}$ ) if $\lambda _{1},\ldots ,\lambda _{l}$ are distinct.

A polynomial weight is a dominant weight $\lambda $ such that $\lambda _{l}\geq 0$ . In other words, polynomial weights of $\operatorname {\mathrm {GL}} _{l}$ are integer partitions of length at most l.

The algebra of virtual $\operatorname {\mathrm {GL}} _{l}$ characters $({\mathbf k} X)^{W}$ can be identified with the algebra of symmetric Laurent polynomials ${\mathbf k} [x_{1}^{\pm 1},\ldots ,x_{l}^{\pm 1}]^{S_{l}}$ . If $\lambda $ is a polynomial weight, the irreducible character $\chi _{\lambda }$ is equal to the Schur function $s_{\lambda }(x_{1},\ldots ,x_{l})$ . Given a virtual $\operatorname {\mathrm {GL}} _{l}$ character $f(x)= f(x_1,\dots ,x_l) = \sum _{\lambda }c_{\lambda }\chi _{\lambda }$ , we denote the partial sum over polynomial weights $\lambda $ by $f(x)_{\operatorname {\mathrm {pol}} }$ (this is different from taking the polynomial terms of $f(x)$ considered as a Laurent polynomial). Thus, $f(x)_{\operatorname {\mathrm {pol}} }$ is a symmetric polynomial in l variables. We also use this notation for infinite formal sums $f(x)$ of irreducible $\operatorname {\mathrm {GL}} _{l}$ characters, in which case $f(x)_{\operatorname {\mathrm {pol}} }$ is a symmetric formal power series.

The Weyl symmetrization operator for $\operatorname {\mathrm {GL}} _{l}$ is

(17) $$ \begin{align} {\boldsymbol \sigma } \, f(x_{1},\ldots,x_{l}) = \sum _{w\in S_{l}} w\left(\frac{f(x)}{\prod _{i<j}(1-x_{j}/x_{i})} \right). \end{align} $$

For dominant weights $\lambda $ , the Weyl character formula can be written $\chi _{\lambda } = {\boldsymbol \sigma } (x^{\lambda })$ .

Fix a weight $\rho $ such that $\langle \alpha _{i}^{\vee } ,\rho \rangle = 1$ for each simple coroot $\alpha _{i}^{\vee } $ , for example, $\rho = (l-1,\ldots ,1,0)$ . Although $\rho $ is only unique up to adding a constant vector, all formulas in which $\rho $ appears will be independent of the choice. Let $\mu $ be any weight, not necessarily dominant. If $\mu +\rho $ is not regular, then ${\boldsymbol \sigma } (x^{\mu }) = 0$ . Otherwise, if $w\in S_{l}$ is the unique permutation such that $w(\mu +\rho )=\lambda +\rho $ for $\lambda $ dominant,

(18) $$ \begin{align} {\boldsymbol \sigma } (x^{\mu }) = (-1)^{\ell(w)}\chi _{\lambda }. \end{align} $$

The following identities are useful for working with the Weyl symmetrization operator. Here and after, $\langle - \rangle \, \Psi $ stands for a coefficient in an expression $\Psi $ , relative to an expansion which will be clear from the context. Thus, in equation (20), $\langle \chi _{\lambda } \rangle $ denotes the coefficient of an irreducible $\operatorname {\mathrm {GL}} _{n}$ character, while $\langle x^{0} \rangle $ denotes the constant term in the variables $x_{i}$ .

Lemma 2.3.1. For any $\operatorname {\mathrm {GL}} _{l}$ weights $\lambda ,\mu $ and Laurent polynomial $\phi (x)=\phi (x_{1},\ldots ,x_{l})$ , we have

(19) $$ \begin{align} \overline{\chi_{\lambda}}\, \prod _{i<j}(1-x_{i}/x_{j}) = \sum _{w\in S_{l}} (-1)^{\ell(w)}x^{-w(\lambda +\rho ) +\rho }, \end{align} $$

(20) $$ \begin{align} \langle \chi _{\lambda } \rangle \, {\boldsymbol \sigma }(\phi (x)) = \langle x^{0} \rangle \, \overline{ \chi _{\lambda }} \, \phi (x)\prod _{i<j} (1-x_{i}/x_{j}), \end{align} $$

(21) $$ \begin{align} {\boldsymbol \sigma } (x^{\mu })_{\operatorname{\mathrm{pol}} } = \langle z^{-\mu } \rangle \Omega [\overline{Z}X] \, \prod _{i<j}(1-z_{i}/z_{j}) \end{align} $$

in alphabets $X = x_{1}+\cdots + x_{l}$ and $Z = z_{1}+\cdots +z_{l}$ , where $\overline {Z} = z_{1}^{-1}+\cdots +z_{l}^{-1}$ .

Proof. Identity (19) follows directly from the Weyl character formula. To prove equation (20), by linearity it suffices to verify the formula when $\phi (x)= x^{\mu }$ is any Laurent monomial. Then using equation (19), the right side becomes $\langle x^{-\mu } \rangle \, \overline { \chi _{\lambda }} \prod _{i<j} (1-x_{i}/x_{j}) = \langle x^{-\mu } \rangle \sum _{w\in S_{l}} (-1)^{\ell (w)}x^{-w(\lambda +\rho ) +\rho }.$ This is $(-1)^{\ell (w)}$ if $\mu +\rho =w(\lambda +\rho )$ , or zero if there is no such w, which agrees with $\langle \chi _{\lambda } \rangle \, {\boldsymbol \sigma }(x^{\mu })$ .

The last identity is then proved from the Cauchy identity (12) followed by equation (20) (applied with z in place of x on the right):

(22) $$ \begin{align} \langle z^{-\mu } \rangle \Omega [\overline{Z}X] \, \prod _{i<j}(1-z_{i}/z_{j}) = \sum_{\lambda} s_{\lambda}(X) \cdot \langle z^{-\mu } \rangle &s_{\lambda}[\overline{Z}] \prod _{i<j}(1-z_{i}/z_{j}) \notag \\[5pt] &= \sum_{\lambda} s_{\lambda}(X) \cdot \langle \chi _{\lambda } \rangle {\boldsymbol \sigma }(x^{\mu}) = {\boldsymbol \sigma } (x^{\mu })_{\operatorname{\mathrm{pol}} }. \end{align} $$

2.4 Hall–Littlewood symmetrization

Given a Laurent polynomial $\phi (x_{1},\ldots ,x_{l})$ , we define

(23) $$ \begin{align} {\mathbf H} _{q}(\phi (x)) = {\boldsymbol \sigma } \Bigl( \frac{\phi (x)}{\prod _{i<j}(1-q\, x_{i}/x_{j})} \Bigr) = \sum _{w\in S_{l}} w\left(\frac{\phi (x)}{\prod _{i<j}((1-x_{j}/x_{i})(1-q\, x_{i}/x_{j}))} \right). \end{align} $$

Here and in similar raising operator formulas elsewhere, the factors $1/(1-q\, x_{i}/x_{j})$ are to be understood as geometric series, making ${\mathbf H} _{q}(\phi (x))$ an infinite formal sum of irreducible $\operatorname {\mathrm {GL}} _{l}$ characters with coefficients in ${\mathbf k} $ . Since $1/(1-q\, x_{i}/x_{j})$ is a power series in q, if we expand the coefficients of $\phi (x)$ as formal Laurent series in q, then ${\mathbf H} _{q}(\phi (x))$ becomes a formal Laurent series in q over virtual $\operatorname {\mathrm {GL}}_{l}$ characters. This is how the last formula in equation (23) should be interpreted.

3.1 Overview

We recall here some definitions and results from [Reference Burban and Schiffmann6, Reference Feigin and Tsymbaliuk8, Reference Negut30, Reference Schiffmann31, Reference Schiffmann and Vasserot32] concerning the elliptic Hall algebra ${\mathcal E} $ of Burban and Schiffmann [Reference Burban and Schiffmann6] (or Schiffmann algebra, for short) and its action on the algebra of symmetric functions $\Lambda $ , constructed by Feigin and Tsymbaliuk [Reference Feigin and Tsymbaliuk8] and Schiffmann and Vasserot [Reference Schiffmann and Vasserot32].

From a certain point of view, this material is unnecessary: Two of the three quantities equated by equations (1) and (2) are defined without reference to the Schiffmann algebra, and we could take ‘shuffle theorem’ to mean the identity between these two, namely

(24) $$ \begin{align} {\mathcal H} _{{\mathbf b} }(x)_{\operatorname{\mathrm{pol}}} = \sum _{\lambda } t^{a(\lambda )}\, q^{\operatorname{\mathrm{dinv}} _{p}(\lambda )}\, {\mathcal G} _{\nu (\lambda )}(x_{1},\ldots,x_{l}; q^{-1}), \end{align} $$

with ${\mathcal H} _{{\mathbf b} }(x)$ as in equation (3). This identity still has the virtue of equating the combinatorial right-hand side, involving Dyck paths and LLT polynomials, with a simple algebraic left-hand side that is manifestly symmetric in q and t. Furthermore, our proof of equation (1) in Theorem 5.5.1 proceeds by combining equation (2) with a proof of equation (24) (via Theorem 5.3.1) that makes no use of the Schiffmann algebra.

What we need the Schiffmann algebra for is to provide the link between our shuffle theorem and the classical and $(km,kn)$ versions. Indeed, the very definition of the algebraic side in the $(km,kn)$ shuffle theorem is $e_{k}[-MX^{m,n}]\cdot 1$ for a certain operator $e_{k}[-MX^{m,n}]\in {\mathcal E} $ , while the classical shuffle theorems refer implicitly to the same operators through the identity $\nabla ^{m} e_{k} = e_{k}[-MX^{m,1}]\cdot 1$ .

In this section, we review what is needed to relate the symmetric functions $\nabla ^{m} e_{k}$ and $({\mathcal H} _{{\mathbf b} })_{\operatorname {\mathrm {pol}} }$ to the action of the elements $e_{k}[-MX^{m,n}]$ and $D_{{\mathbf b} }$ on $\Lambda $ . For ease of reference, we have collected most of the statements that will be used elsewhere in the paper in §3.7, after the technical development in §§3.2–3.6.

3.2 The algebra ${\mathcal E} $

Let ${\mathbf k} ={\mathbb Q} (q,t)$ . The Schiffmann algebra ${\mathcal E} $ is generated by subalgebras $\Lambda (X^{m,n})$ isomorphic to the algebra $\Lambda _{{\mathbf k} }$ of symmetric functions, one for each pair of coprime integers $(m,n)$ , and a central Laurent polynomial subalgebra ${\mathbf k} [c_{1}^{\pm 1}, c_{2}^{\pm 1}]$ . A full presentation of ${\mathcal E} $ in our notation can be found in [Reference Blasiak, Haiman, Morse, Pun and Seelinger5, §3.2]. Here, we only use a few of the defining relations, invoking them where needed.

For purposes of comparison with [Reference Burban and Schiffmann6, Reference Schiffmann31, Reference Schiffmann and Vasserot32], our notation (on the left-hand side of each formula) is related as follows to that in [Reference Burban and Schiffmann6, Definition 6.4] (on the right-hand side). Note that our indices $(m,n)\in {\mathbb Z} ^{2}$ correspond to transposed indices $(n,m)$ in [Reference Burban and Schiffmann6].

(25) $$ \begin{align} \begin{gathered} q = \sigma ^{-1},\quad t = \bar{\sigma }^{-1},\\ c_{1}^{m}c_{2}^{n} = \kappa _{n,m}^{-2},\\ \omega p_{k}(X^{m,n}) = \kappa _{kn,km}^{\varepsilon _{n,m}} u_{kn,km},\\ e_{k}[-\widehat{M} X^{m,n}] = \kappa _{kn,km}^{\varepsilon _{n,m}} \theta _{kn,km}, \end{gathered} \end{align} $$

where $\varepsilon _{n,m}$ , which is equal to $(1-\epsilon _{n,m})/2$ in the notation of [Reference Burban and Schiffmann6], is given by

(26) $$ \begin{align} \varepsilon _{n,m} = \begin{cases} 1& n<0 \text{ or } (m,n)=(-1,0)\\ 0& \text{otherwise}. \end{cases} \end{align} $$

The expression $e_{k}[-\widehat {M} X^{m,n}]$ in equation (25) uses plethystic substitution (§2.2) with

(27) $$ \begin{align} \widehat{M} = (1-\frac{1}{q\, t})M \quad \text{where}\quad M = (1-q)(1-t). \end{align} $$

The quantity M will be referred to repeatedly.

3.3 Action of ${\mathcal E} $ on $\Lambda $

A natural action of ${\mathcal E} $ by operators on $\Lambda (X)$ has been constructed in [Reference Feigin and Tsymbaliuk8, Reference Schiffmann and Vasserot32]. Actually, these references give several different normalizations of essentially the same action. The action we use is a slight variation of the action in [Reference Schiffmann and Vasserot32, Theorem 3.1].

To write it down, we need to recall some notions from the theory of Macdonald polynomials. Let $\tilde {H} _{\mu }(X;q,t)$ denote the modified Macdonald polynomials [Reference Garsia and Haiman11], which can be defined in terms of the integral form Macdonald polynomials $J_{\mu }(X;q,t)$ of [Reference Macdonald26] by

(28) $$ \begin{align} \tilde{H} _{\mu }(X;q,t) = t^{n(\mu )} J_{\mu }[\frac{X}{1-t^{-1}};q,t^{-1}], \end{align} $$

with $n(\mu )$ as in equation (6). For any symmetric function $f\in \Lambda $ , let $f[{\mathbf B} ]$ , $f[\overline {{\mathbf B} }]$ denote the eigenoperators on the basis $\{\tilde {H_{\mu }} \}$ of $\Lambda $ such that

(29) $$ \begin{align} f[{\mathbf B} ]\, \tilde{H} _{\mu } = f[B_{\mu }(q,t)]\, \tilde{H} _{\mu },\quad f[\overline{{\mathbf B} }]\, \tilde{H} _{\mu } = f[\overline{B_{\mu }(q,t)}]\, \tilde{H} _{\mu } \end{align} $$

with $B_{\mu }(q,t)$ as in equation (8) and $\overline {B_{\mu }(q,t)} = B_{\mu }(q^{-1},t^{-1})$ . More generally, we use the overbar to signify inverting the variables in any expression, for example,

(30) $$ \begin{align} \overline{M} = (1-q^{-1})(1-t^{-1}). \end{align} $$

The next proposition essentially restates the contents of [Reference Schiffmann and Vasserot32, Theorem 3.1 and Proposition 4.10] in our notation. To be more precise, since these two theorems refer to different actions of ${\mathcal E} $ on $\Lambda $ , one must first use the plethystic transformation in [Reference Schiffmann and Vasserot32, §4.4] to express [Reference Schiffmann and Vasserot32, Proposition 4.10] in terms of the action in [Reference Schiffmann and Vasserot32, Theorem 3.1]. Rescaling the generators $u_{\pm 1,l}$ then yields the following.

We will show in Proposition 3.3.4, below, that the elements $p_{1}[-MX^{1,a}]\in {\mathcal E} $ act on $\Lambda $ as operators $D_{a}$ given by the coefficients $D_{a} = \langle z^{-a} \rangle D(z)$ of a generating series

(34) $$ \begin{align} D(z) = \sum _{a\in {\mathbb Z} } D_{a} z^{-a} \end{align} $$

defined by either of the equivalent formulas

(35) $$ \begin{align} D(-z) = \Omega [-z^{-1}X]^{\bullet } \, \Omega [z M X]^{\perp }\quad \text{or}\quad D(z) = (\omega \Omega [z^{-1}X])^{\bullet }\, (\omega \Omega [-z M X])^{\perp }, \end{align} $$

using the operator notation from §2.1. These operators $D_{a}$ differ by a sign $(-1)^{a}$ from those studied in [Reference Bergeron, Garsia, Haiman and Tesler3, Reference Garsia, Haiman and Tesler12] and by a plethystic transformation from operators previously introduced by Jing [Reference Jing23].

Lemma 3.3.3. We have the identities

(36) $$ \begin{align} [\, (\omega p_{k}[-X/M])^{\bullet },\, D_{a}\, ] = -D_{a+k},\quad [\, (\omega p_{k}(X))^{\perp },\, D_{a}\, ] = D_{a-k}. \end{align} $$

Proof. We start with the second identity, which is equivalent to

(37) $$ \begin{align} [\, (\omega p_{k}(X))^{\perp },\, D(z)\, ] = z^{-k}D(z). \end{align} $$

Since all operators of the form $f(X)^{\perp }$ commute with each other, equation (37) follows from the definition of $D(z)$ and

(38) $$ \begin{align} [\, (\omega p_{k}(X))^{\perp },\, (\omega \Omega [z^{-1}X])^{\bullet } \,] = z^{-k} (\omega \Omega [z^{-1}X])^{\bullet }. \end{align} $$

To verify the latter identity, note first that equation (15) and $\Omega [z^{-1}X] = \exp \sum _{k>0} p_{k} z^{-k}/k$ imply

(39) $$ \begin{align} [\, p_{k}(X)^{\perp },\, \Omega [z^{-1}X]^{\bullet }\, ] = z^{-k} \, \Omega [z^{-1}X]^{\bullet }. \end{align} $$

Conjugating both sides by $\omega $ and using $(\omega f)^{\bullet } = \omega \cdot f^{\bullet }\cdot \omega $ and $(\omega f)^{\perp } = \omega \cdot f^{\perp }\cdot \omega $ gives equation (38).

For the first identity in equation (36), consider the modified inner product defined by

(40) $$ \begin{align} \langle f,g \rangle' = \langle f[-MX],g \rangle = \langle f,g[-MX] \rangle. \end{align} $$

The second equality here, which shows that $\langle -,- \rangle '$ is symmetric, follows from orthogonality of the power-sums $p_{\lambda }$ . For any f, the operators $f^{\perp }$ and $f[-X/M]^{\bullet }$ are adjoint with respect to $\langle -,- \rangle '$ . Using this and the definition of $D(z)$ , we see that $D(z^{-1})$ is the $\langle -,- \rangle '$ adjoint of $D(z)$ ; hence, $D_{-a}$ is adjoint to $D_{a}$ . Taking adjoints on both sides of the second identity in equation (36) now implies the first.

Proof. It is known [Reference Haiman18, Proposition 2.4] that $D_{0}\tilde {H} _{\mu } = (1-MB_{\mu })\tilde {H} _{\mu }$ . From Proposition 3.3.1 (ii), we see that $p_{1}[-M X^{1,0}]$ acts by the same operator, giving the case $a=0$ .

Among the defining relations of ${\mathcal E} $ are

(41) $$ \begin{align} [\, \omega p_{k}(X^{0,1}),\, p_{1}(X^{1,a})\, ] = - p_{1}(X^{1,a+k}),\quad [\, \omega p_{k}(X^{0,-1}),\, p_{1}(X^{1,a})\, ] = p_{1}(X^{1,a-k}). \end{align} $$

Multiplying these by $-M$ and using Proposition 3.3.1 (iii) to compare with equation (36) reduces the general result to the case $a=0$ .

3.4 The operator $\nabla $

As in [Reference Bergeron, Garsia, Haiman and Tesler3], we define an eigenoperator $\nabla $ on the Macdonald basis by

(42) $$ \begin{align} \nabla \tilde{H} _{\mu } = t^{n(\mu )}q^{n(\mu ^{*})} \tilde{H} _{\mu }, \end{align} $$

with $n(\mu )$ as in equation (6). Since $t^{n(\mu )}q^{n(\mu ^{*})} = e_{n}[B_{\mu }(q,t)]$ for $n = |\mu |$ , we see that $\nabla $ coincides in degree n with $e_{n}[{\mathbf B} ]$ . Although the operators $e_{n}[{\mathbf B} ]$ belong to ${\mathcal E} $ acting on $\Lambda $ , the operator $\nabla $ does not. Its role, rather, is to internalize a symmetry of this action.

Lemma 3.4.1. Conjugation by the operator $\nabla $ provides a symmetry of the action of ${\mathcal E} $ on $\Lambda $ , namely

(43) $$ \begin{align} \nabla\, f(X^{m,n})\, \nabla^{-1} = f(X^{m+n,n}). \end{align} $$

Proof. For $m=\pm 1$ , $n=0$ , this says that $\nabla $ commutes with the other Macdonald eigenoperators, which is clear.

It is known from [Reference Burban and Schiffmann6] that the group of ${\mathbf k} $ -algebra automorphisms of ${\mathcal E} $ includes one which acts on the subalgebras $\Lambda (X^{m,n})$ by $f(X^{m,n})\mapsto f(X^{m+n,n})$ and on the central subalgebra ${\mathbf k} [c_{1}^{\pm 1},c_{2}^{\pm 1}]$ by an automorphism which fixes the central character in Proposition 3.3.1 (i).

The $\Lambda (X^{m,n})$ for $n>0$ are all contained in the subalgebra of ${\mathcal E} $ generated by the elements $p_{1}(X^{a,1})$ . To prove equation (43) for $n>0$ , it therefore suffices to verify the operator identity $\nabla \, p_{1}(X^{a,1})\, \nabla ^{-1} = p_{1}(X^{a+1,1})$ .

In ${\mathcal E} $ , there are relations

(44) $$ \begin{align} [\, \omega p_{k}(X^{1,0}),\, p_{1}(X^{a,1})\, ] = p_{1}(X^{a+k,1}),\quad [\, c_{1}^{-k}\omega p_{k}(X^{-1,0}),\, p_{1}(X^{a,1})\, ] = -p_{1}(X^{a-k,1}). \end{align} $$

Since $\nabla $ commutes with the action of $\Lambda (X^{\pm 1,0})$ , these relations reduce the problem to the case $a=0$ , that is, to the identity $\nabla \, p_{1}(X^{0,1})\, \nabla ^{-1} = p_{1}(X^{1,1})$ . By Propositions 3.3.1 and 3.3.4, this is equivalent to the operator identity $\nabla \cdot p_{1}(X)^{\bullet }\cdot \nabla ^{-1} = D_{1}$ , which is [Reference Bergeron, Garsia, Haiman and Tesler3, I.12 (iii)].

We leave the similar argument for the case $n<0$ , using [Reference Bergeron, Garsia, Haiman and Tesler3, I.12 (iv)], to the reader.

3.5 Shuffle algebra

The operators of interest to us belong to the ‘right half-plane’ subalgebra ${\mathcal E} ^{+}\subseteq {\mathcal E} $ generated by the $\Lambda (X^{m,n})$ for $m>0$ , or equivalently by the elements $p_{1}(X^{1,a})$ . The subalgebra ${\mathcal E} ^{+}$ acts on $\Lambda $ as the algebra generated by the operators $D_{a}$ . It was shown in [Reference Schiffmann and Vasserot32] that ${\mathcal E} ^{+}$ is isomorphic to the shuffle algebra constructed in [Reference Feigin and Tsymbaliuk8] and studied further in [Reference Negut30], whose definition we now recall.

We fix the rational function

(45) $$ \begin{align} \Gamma (x/y) = \frac{1-q\, t\, x/y}{(1-y/x)(1-q\, x/y)(1-t\, x/y)}, \end{align} $$

and define, for each l, a $q,t$ analog of the symmetrization operator ${\mathbf H} _{q}$ in equation (23) by

(46) $$ \begin{align} {\mathbf H} _{q,t}(\phi (x_{1},\ldots,x_{l})) = \sum _{w\in S_{l}} w &\biggl( \phi(x) \cdot \prod _{i<j} \Gamma \bigl(\frac{x_{i}}{x_{j}}\bigr) \biggr) = {\boldsymbol \sigma } \left(\frac{\phi (x)\, \prod _{i<j} (1 - q\, t\, x_{i}/x_{j})}{\prod _{i<j} ((1 - q\, x_{i}/x_{j}) (1 - t\, x_{i}/x_{j}))} \right). \end{align} $$

We write ${\mathbf H} _{q,t}^{(l)}$ when we want to make the number of variables explicit.

Let $T = T({\mathbf k} [x^{\pm 1}])$ be the tensor algebra on the Laurent polynomial ring ${\mathbf k} [x^{\pm 1}]$ in one variable, that is, the noncommutative polynomial algebra with generators corresponding to the basis elements $x^{a}$ of ${\mathbf k} [x^{\pm 1}]$ as a vector space. Identifying $T^{k} = T^{k}({\mathbf k} [x^{\pm 1}])$ with ${\mathbf k} [x_{1}^{\pm 1},\ldots ,x_{k}^{\pm 1}]$ , the product in T is given by ‘concatenation,’

(47) $$ \begin{align} f\cdot g = f(x_{1},\ldots,x_{k})g(x_{k+1},\ldots,x_{k+l}),\quad \text{for } f\in T^{k}, g\in T^{l}. \end{align} $$

For each l, let $I^{l}\subseteq T^{l}$ be the kernel of the symmetrization operator ${\mathbf H} _{q,t}^{(l)}$ . Since ${\mathbf H} _{q,t}^{(l)}$ factors through the operator ${\mathbf H} _{q,t}^{(k)}$ in any k consecutive variables $x_{i+1},\ldots ,x_{i+k}$ , the graded subspace $I = \bigoplus _{l}I^{l}\subseteq T$ is a two-sided ideal. The shuffle algebra is defined to be the quotient $S = T/I$ . Note that S is generated by its tensor degree $1$ component $S^{1}$ by definition. We will not use the second, larger, type of shuffle algebra that was also introduced in [Reference Feigin and Tsymbaliuk8, Reference Negut30].

For clarity, we note that the factor $-M$ in $p_{1}[-MX^{1,a}] = -M p_{1}(X^{1,a})$ makes no difference to the statement but is a convenient normalization for us since it makes $\psi (x_{1}^{a_{1}}\cdots x_{l}^{a_{l}})$ act on $\Lambda $ as $D_{a_{1}}\cdots D_{a_{l}}$ . We also note that our $\Gamma (x/y)$ differs by a factor symmetric in $x,y$ from the function $g(y/x)$ in [Reference Schiffmann and Vasserot32, (10.3)], which makes the product in our shuffle algebra S opposite to that of the algebra ${\mathbf S} $ in [Reference Schiffmann and Vasserot32]. This is as it should be since the isomorphism in [Reference Schiffmann and Vasserot32] is from ${\mathbf S} $ to the ‘upper half-plane’ subalgebra of ${\mathcal E} $ generated by the elements $p_{1}(X^{a,1})$ , and the symmetry $p_{1}(X^{a,1})\leftrightarrow p_{1}(X^{1,a})$ is an antihomomorphism.

By construction, Laurent polynomials $\phi (x), \phi '(x)$ in variables $x_{1},\ldots ,x_{l}$ define the same element of S or, equivalently, map via $\psi $ to the same element of ${\mathcal E} ^{+}$ , if and only if ${\mathbf H} _{q,t}(\phi ) = {\mathbf H} _{q,t}(\phi ')$ . We can regard ${\mathbf H} _{q,t}(\phi )$ as an infinite formal sum of $\operatorname {\mathrm {GL}} _{l}$ characters with coefficients in ${\mathbf k} $ , in the same manner as for ${\mathbf H} _{q}$ . Representing elements of S, or of ${\mathcal E} ^{+}$ , in this way leads to the following useful formula for describing their action on $1\in \Lambda $ .

Proposition 3.5.2. Given a Laurent polynomial $\phi =\phi (x_{1},\ldots ,x_{l})$ , let $\zeta = \psi (\phi ) \in {\mathcal E} ^{+}$ be its image under the isomorphism in Proposition 3.5.1. Then with the action of ${\mathcal E} $ on $\Lambda $ given by Proposition 3.3.1, we have

(48) $$ \begin{align} \omega (\zeta \cdot 1)(x_{1},\ldots,x_{l}) = {\mathbf H} _{q,t}(\phi)_{\operatorname{\mathrm{pol}} }. \end{align} $$

Moreover, the Schur function expansion of the symmetric function $\omega (\zeta \cdot 1)(X)$ contains only terms $s_{\lambda }$ with $\ell (\lambda )\leq l$ , so equation (48) determines $\zeta \cdot 1$ .

Proof. It suffices to consider the case when $\phi (x)=x^{{\mathbf a}} = x_{1}^{a_{1}}\cdots x_{l}^{a_{l}}$ and thus (by Proposition 3.3.4) $\zeta $ acts on $\Lambda $ as the operator $D_{a_{1}}\cdots D_{a_{l}}$ . To find $\zeta \cdot 1$ , we use equation (16) to compute

(49) $$ \begin{align} D(z_{1})\cdots D(z_{l}) =\bigl(\prod _{i<j} \Omega [-z_{i}/z_{j} M] \bigr)\, (\omega \Omega [\overline{Z} X])^{\bullet }\, (\omega \Omega [-Z M X])^{\perp }, \end{align} $$

where $Z = z_{1}+\cdots +z_{l}$ . Acting on $1$ , applying $\omega $ , and taking the coefficient of $z^{-{\mathbf a} }$ gives

(50) $$ \begin{align} \omega (\zeta \cdot 1)(X) = \langle z^{-{\mathbf a} } \rangle &\bigl(\prod _{i<j} \Omega [-z_{i}/z_{j} M] \bigr)\, \Omega [\overline{Z} X] \notag \\ &\quad = \langle z^{0} \rangle \bigl(z^{{\mathbf a} }\prod _{i<j} \frac{1-q\, t\, z_{i}/z_{j}}{(1-q \, z_{i}/z_{j})(1-t \, z_{i}/z_{j})} \bigr)\, \Omega [\overline{Z} X] \, \prod _{i<j}(1-z_{i}/z_{j}). \end{align} $$

Since Z has l variables, this implies that all Schur functions $s_{\lambda } $ in $\omega (\zeta \cdot 1)(X)$ have $\ell (\lambda )\leq l$ . Identity (48) for $\phi (x) = x^{{\mathbf a} }$ follows by specializing X to $x_1+\cdots +x_l$ and applying equation (21).

3.6 Distinguished elements

Given a rational function $\phi (x_{1},\ldots ,x_{l})$ , it may happen that we have an identity of rational functions ${\mathbf H} _{q,t}(\phi ) = {\mathbf H} _{q,t}(\eta )$ for some Laurent polynomial $\eta (x)$ . In this case, ${\mathbf H} _{q,t}(\phi )$ is the representative of the image of $\eta $ in S, or of $\psi (\eta )\in {\mathcal E} ^{+}$ , even though $\phi (x)$ is not necessarily a Laurent polynomial. For the shuffle algebra S under consideration here, Negut [Reference Negut30, Proposition 6.1] showed that this happens for

(51) $$ \begin{align} \phi (x) = \frac{x_{1}^{b_{1}}\cdots x_{l}^{b_{l}}}{\prod _{i=1}^{l-1}(1-q\, t\, x_{i}/x_{i+1})}. \end{align} $$

Accordingly, there are distinguished elements

(52) $$ \begin{align} D_{b_{1},\ldots,b_{l}} = \psi (\eta )\in {\mathcal E} ^{+}, \end{align} $$

where $\psi \colon S \rightarrow {\mathcal E} ^{+}$ is the isomorphism in Proposition 3.5.1 and $\eta (x)$ is any Laurent polynomial such that ${\mathbf H} _{q,t}(\phi ) = {\mathbf H} _{q,t}(\eta )$ for the function $\phi $ in equation (51). Although there seems to be no particularly nice preferred choice for $\eta $ , we can avoid working directly with $\eta $ by using the element $\phi $ from equation (51) in equation (48).

Negut identified certain of the elements $D_{b_{1},\ldots ,b_{l}}$ as (in our notation) ribbon skew Schur functions $s_{R}[-MX^{m,n}]$ . The following result is a special case.

Proposition 3.6.1 [Reference Negut30, Proposition 6.7].

Let m, k be positive integers and n any integer with $m,n$ coprime. For $i=1,\ldots ,km$ , let $b_{i} = \lceil i n/m \rceil -\lceil (i-1) n/m \rceil $ ; if $n \geq 0$ , this is the number of south steps at $x=i-1$ in the highest south-east lattice path $\delta $ weakly below the line from $(0,kn)$ to $(km,0)$ . Then

(53) $$ \begin{align} e_{k}[-M X^{m,n}] = D_{b_{1},\ldots,b_{km}}. \end{align} $$

Lemma 3.6.2. For any indices $b_{1},\ldots ,b_{l}$ , we have

(54) $$ \begin{align} D_{b_{1},\ldots,b_{l},0} \cdot 1 = D_{b_{1},\ldots,b_{l}} \cdot 1. \end{align} $$

Proof. Using Proposition 3.5.2, each side of equation (54) is characterized by its evaluation

(55) $$ \begin{align} \omega (D_{b_{1},\ldots,b_{l}} \cdot 1)(x_{1},\ldots,x_{l}) = {\mathbf H} ^{(l)}_{q,t}\Bigl(\frac{x_{1}^{b_{1}}\cdots x_{l}^{b_{l}}}{\prod _{i=1}^{l-1}(1 - q\, t\, x_{i}/x_{i+1})}\Bigr)_{\operatorname{\mathrm{pol}} } \end{align} $$

(56) $$ \begin{align} \omega (D_{b_{1},\ldots,b_{l},0} \cdot 1)(x_{1},\ldots,x_{l+1}) = {\mathbf H} ^{(l+1)}_{q,t}\Bigl(\frac{x_{1}^{b_{1}}\cdots x_{l}^{b_{l}}}{(1 - q\, t\, x_{l}/x_{l+1})\prod _{i=1}^{l-1}(1 - q\, t\, x_{i}/x_{i+1})}\Bigr)_{\operatorname{\mathrm{pol}} }. \end{align} $$

Terms with a negative exponent of $x_{l+1}$ inside the parenthesis in equation (56) contribute zero after we apply ${\mathbf H} _{q,t}(-)_{\operatorname {\mathrm {pol}} }$ . We can therefore drop all but the constant term of the geometric series factor $1/(1 - q\, t\, x_{l}/x_{l+1})$ since the other factors are independent of $x_{l+1}$ . This shows that the right-hand side of equation (56) is the same as in equation (55), except that it has ${\mathbf H} ^{(l+1)}_{q,t}$ in place of ${\mathbf H} ^{(l)}_{q,t}$ .

It follows that $\omega (D_{b_{1},\ldots ,b_{l},0} \cdot 1)$ is a linear combination of Schur functions $s_{\lambda }(X)$ with $\ell (\lambda )\leq l$ , and that $\omega (D_{b_{1},\ldots ,b_{l},0} \cdot 1)$ and $\omega (D_{b_{1},\ldots ,b_{l}} \cdot 1)$ evaluate to the same symmetric function in l variables. Hence, they are identical.

3.7 Summary

Most of what we use from this section in other parts of the paper can be summarized as follows.

Definition 3.7.1. Given ${\mathbf b} = (b_{1},\ldots ,b_{l})\in {\mathbb Z} ^{l}$ , the infinite series of $\operatorname {\mathrm {GL}} _{l}$ characters ${\mathcal H} _{{\mathbf b} }(x) = {\mathcal H} _{b_{1},\ldots ,b_{l}}(x_{1},\ldots ,x_{l})$ is defined by

(57) $$ \begin{align} {\mathcal H} _{{\mathbf b} }(x) = {\mathbf H} _{q,t}\Bigl(\frac{x^{{\mathbf b} }}{\prod _{i=1}^{l-1}(1-q\, t\, x_{i}/x_{i+1})}\Bigr) = {\mathbf H} _{q}\Bigl( x^{{\mathbf b} }\, \frac{\prod _{i+1<j}(1-q\, t\, x_{i}/x_{j})}{\prod _{i<j}(1-t\, x_{i}/x_{j})} \Bigr), \end{align} $$

where ${\mathbf H} _{q,t}$ is given by equation (46) and ${\mathbf H} _{q}$ by equation (23) or where ${\mathcal H} _{{\mathbf b} }(x)$ is given in fully expanded form by equation (3).

In terms of this definition, we have the following special case of Proposition 3.5.2, which was stated as identity (2) in the introduction.

Corollary 3.7.2. For the Negut element $D_{{\mathbf b} }\in {\mathcal E} $ acting on $\Lambda $ , the symmetric function $\omega (D_{{\mathbf b} }\cdot 1)$ evaluated in l variables is given by

(58) $$ \begin{align} \omega (D_{{\mathbf b} } \cdot 1)(x_{1},\ldots,x_{l}) = {\mathcal H} _{{\mathbf b} }(x)_{\operatorname{\mathrm{pol}} }\,. \end{align} $$

Moreover, all terms $s_{\lambda }$ in the Schur expansion of $\omega (D_{{\mathbf b} } \cdot 1)(X)$ have $\ell (\lambda )\leq l$ , so $\omega (D_{{\mathbf b} } \cdot 1)$ is determined by its evaluation in l variables.

In the special cases where the index ${\mathbf b} $ is the sequence of south runs in the highest $(km,kn)$ Dyck path, $D_{{\mathbf b} }\cdot 1$ can also be expressed as follows.

Corollary 3.7.3. For $i = 1,\ldots ,l=km+1$ , let $b_{i}$ be the number of south steps at $x = i-1$ in the highest south-east lattice path $\delta $ weakly below the line from $(0,kn)$ to $(km,0)$ , including $b_{l}=0$ . Then the Negut element $D_{b_{1},\ldots ,b_{l}}$ and the operator $e_{k}[-MX^{m,n}]$ agree when applied to $1\in \Lambda $ , that is, we have

(59) $$ \begin{align} D_{b_{1},\ldots,b_{l}} \cdot 1 = e_{k}[-MX^{m,n}]\cdot 1. \end{align} $$

Corollary 3.7.4 (also proven in [Reference Bergeron, Garsia, Sergel Leven and Xin4]).

In the case $n=1$ of equation (59), we further have

(60) $$ \begin{align} \nabla ^{m} e_{k}(X) = e_{k}[-MX^{m,1}]\cdot 1. \end{align} $$

Remark 3.7.5. Equations (46), (54), (57), (58), (59) and (60) imply the raising operator formula

(61) $$ \begin{align} (\omega\nabla^m e_{k})(x_1,\dots,x_l) = {\boldsymbol \sigma } \left(\frac{x_1 x_{m+1} x_{2m+1}\cdots x_{(k-1)m+1} \, \prod _{i+1<j} (1 - q\, t\, x_{i}/x_{j})}{\prod _{i<j} ((1 - q\, x_{i}/x_{j}) (1 - t\,x_{i}/x_{j}))} \right)_{\operatorname{\mathrm{pol}}}, \end{align} $$

provided $l \geq (k-1)m+1$ .

4.1 Combinatorial LLT polynomials

The content of a box $a=(x,y)$ in row y, column x of any skew diagram is $c(a)=x-y$ .

Let $\nu = (\nu ^{(1)},\ldots ,\nu ^{(k)})$ be a tuple of skew diagrams. When referring to boxes of $\nu $ , we identify $\nu $ with the disjoint union of the $\nu ^{(j)}$ . Fix $\epsilon>0$ small enough that $k\, \epsilon <1$ . The adjusted content of a box $a\in \nu ^{(j)}$ is $\tilde {c} (a) = c(a)+j\, \epsilon $ . A reading order is any total ordering of the boxes $a\in \nu $ on which $\tilde {c} (a)$ is increasing. In other words, the reading order is lexicographic, first by content, then by the index j for which $a\in \nu ^{(j)}$ , with boxes of the same content in the same component $\nu ^{(j)}$ ordered arbitrarily.

Boxes $a,b\in \nu $ attack each other if $0<|\tilde {c} (a)-\tilde {c} (b)|<1$ . If $a\in \nu ^{(i)}$ precedes $b\in \nu ^{(j)}$ in the reading order, the attacking condition means that either $c(a)=c(b)$ and $i<j$ or $c(b)=c(a)+1$ and $i>j$ . We also say that $a,b$ form an attacking pair in $\nu $ .

By a semistandard Young tableau on the tuple $\nu $ , we mean a map $T\colon \nu \rightarrow {\mathbb Z} _{>0} $ which restricts to a semistandard tableau on each component $\nu ^{(j)}$ . We write $\operatorname {\mathrm {SSYT}} (\nu )$ for the set of these. The weight of $T\in \operatorname {\mathrm {SSYT}} (\nu )$ is $x^{T} = \prod _{a\in \nu }x_{T(a)}$ . An attacking inversion in T is an attacking pair $a,b$ such that $T(a)>T(b)$ , where a precedes b in the reading order. We define $\operatorname {\mathrm {inv}} (T)$ to be the number of attacking inversions in T.

Example 4.1.1. The picture below shows a tuple of skew diagrams $\nu = (\nu ^{(1)}, \nu ^{(2)})$ , with dashed lines indicating boxes of equal content and boxes numbered in reading order, along with a semistandard tableau T on $\nu $ .

(62)

The tuple $\nu $ contains $7$ attacking pairs $(a,b)$ : In the numbering shown, these are $(2,3)$ , $(3,4)$ , $(4,5)$ , $(4,6)$ , $(5,7)$ , $(6,7)$ , $(7,8)$ . The pairs numbered $(3,4)$ , $(4,5)$ and $(6,7)$ form inversions in T, with entries $(T(a),T(b))$ equal to $(4,2)$ , $(2,1)$ and $(4,3)$ , respectively.

Definition 4.1.2. The combinatorial LLT polynomial indexed by a tuple of skew diagrams $\nu $ is the generating function

(63) $$ \begin{align} {\mathcal G}_{\nu }(X;q) = \sum _{T\in \operatorname{\mathrm{SSYT}} (\nu )}q^{\operatorname{\mathrm{inv}} (T)}x^{T}. \end{align} $$

In [Reference Haglund, Haiman, Loehr, Remmel and Ulyanov16], it was shown that ${\mathcal G}_{\nu }(X;q^{-1})$ coincides up to a factor $q^{e}$ with a ribbon tableau LLT polynomial as defined in [Reference Lascoux, Leclerc and Thibon24] and is therefore a symmetric function. A direct and more elementary proof that ${\mathcal G}_{\nu }(X;q)$ is symmetric was given in [Reference Haglund, Haiman and Loehr15].

It is useful in working with the LLT polynomials ${\mathcal G} _{\nu }(X;q)$ to consider a more general combinatorial formalism, as in [Reference Haglund, Haiman and Loehr15, §10]. Let ${\mathcal A} = {\mathcal A} _{+} \coprod {\mathcal A} _{-}$ be a ‘signed’ alphabet with a positive letter $v\in {\mathcal A} _{+}$ and a negative letter $\overline {v}\in {\mathcal A} _{-}$ for each $v\in {\mathbb Z} _{>0} $ and an arbitrary total ordering on ${\mathcal A} $ .

A super tableau on a tuple of skew shapes $\nu $ is a map $T\colon \nu \rightarrow {\mathcal A} $ , weakly increasing along rows and columns, with positive letters increasing strictly on columns and negative letters increasing strictly on rows. A usual semistandard tableau is thus the same thing as a super tableau with all entries positive. Let $\operatorname {\mathrm {SSYT}} _{\pm }(\nu )$ denote the set of super tableaux on $\nu $ .

An attacking inversion in a super tableau is an attacking pair $a,b$ , with a preceding b in the reading order, such that either $T(a)>T(b)$ in the ordering on ${\mathcal A} $ , or $T(a)=T(b) = \overline {v}$ with $\overline {v}$ negative. As before, $\operatorname {\mathrm {inv}} (T)$ denotes the number of attacking inversions.

Lemma 4.1.3 [Reference Haglund, Haiman and Loehr15, (81–82) and Proposition 4.2].

We have the identity

(64) $$ \begin{align} \omega _{Y} {\mathcal G} _{\nu }[X+Y;q] = \sum _{T\in \operatorname{\mathrm{SSYT}} _{\pm }(\nu )} q^{\operatorname{\mathrm{inv}} (T)} x^{T_{+}} y^{T_{-}}, \end{align} $$

where the weight is given by

(65) $$ \begin{align} x^{T_{+}} y^{T_{-}} = \prod _{a\in \nu }\begin{cases} x_{i},& T(a) = i\in {\mathcal A} _{+},\\ y_{i},& T(a) = \overline{i} \in {\mathcal A} _{-}. \end{cases} \end{align} $$

This holds for any choice of the ordering on the signed alphabet ${\mathcal A} $ .

Corollary 4.1.4. We have

(66) $$ \begin{align} \omega\, {\mathcal G} _{\nu }(X;q) = \sum _{T\in \operatorname{\mathrm{SSYT}}_{-}(\nu)} q^{\operatorname{\mathrm{inv}} (T)}x^{T}, \end{align} $$

where the sum is over super tableaux T with all entries negative, and we abbreviate $x^{T_{-}}$ to $x^{T}$ in this case.

Proposition 4.1.6. With $\nu ^{R}$ as in Definition 4.1.5, we have

(67) $$ \begin{align} \omega \, {\mathcal G} _{\nu }(X;q) = q^{I(\nu )}\, {\mathcal G} _{\nu ^{R}}(X;q^{-1}), \end{align} $$

where $I(\nu )$ is the total number of attacking pairs in $\nu $ .

Example 4.1.7. Consider a tuple $\nu = (\nu ^{(1)},\ldots ,\nu ^{(k)})$ in which each skew shape is a column so that $\nu ^R$ is a tuple of rows. The super tableau of shape $\nu ^R$ with all entries a positive letter $1$ has no inversions, whereas the super tableau T of shape $\nu $ with all entries a negative letter $\overline {1}$ has

(68) $$ \begin{align} \operatorname{\mathrm{inv}}(T)=I(\nu), \end{align} $$

where $I(\nu )$ is the total number of attacking pairs in $\nu $ .

4.2 Reminder on Hecke algebras

We recall, in the case of $\operatorname {\mathrm {GL}} _{l}$ , the Hecke algebra action on the group algebra of the weight lattice, as in Lusztig [Reference Lusztig25] or Macdonald [Reference Macdonald27] and due originally to Bernstein and Zelevinsky.

For $\operatorname {\mathrm {GL}} _{l}$ , we identify the group algebra ${\mathbf k} X$ of the weight lattice $X = {\mathbb Z} ^{l}$ with the Laurent polynomial algebra ${\mathbf k} [x_{1}^{\pm 1},\ldots ,x_{l}^{\pm 1}]$ . Here, ${\mathbf k} $ is any ground field containing ${\mathbb Q} (q)$ .

The Demazure–Lusztig operators

(69) $$ \begin{align} T_{i} = q s_{i} + (1-q)\frac{1}{1-x_{i+1}/x_{i}} (s_{i}-1) \end{align} $$

for $i = 1,\ldots ,l-1$ generate an action of the Hecke algebra of $S_{l}$ on ${\mathbf k} [x_{1}^{\pm 1},\ldots ,x_{l}^{\pm 1}]$ . We have normalized the generators so that the quadratic relations are $(T_{i}-q)(T_{i}+1)=0$ . The elements $T_w$ , defined by $T_w = T_{i_1} T_{i_2}\cdots T_{i_m}$ for any reduced expression $w=s_{i_1}s_{i_2}\cdots s_{i_m}$ , form a ${\mathbf k}$ -basis of the Hecke algebra, as w ranges over $S_l$ .

Let $R_{+}$ be the set of positive roots and $Q_{+} = {\mathbb N} R_{+}$ the cone they generate in the root lattice Q. For dominant weights, we define $\lambda \leq \mu $ if $\mu -\lambda \in Q_{+}$ . For polynomial weights of $\operatorname {\mathrm {GL}}_{l}$ , this coincides with the standard partial ordering (7) on partitions.

For any weight $\lambda $ , let $\lambda _{+}$ denote the dominant weight in the orbit $S_{l}\, \lambda $ .

Let $\operatorname {\mathrm {conv}} (S_{l}\, \lambda )$ be the convex hull of $S_{l}\, \lambda $ in the coset $\lambda +Q$ of the root lattice, that is, the set of weights that occur with nonzero multiplicity in the irreducible character $\chi _{\lambda _{+}}$ . Note that $\operatorname {\mathrm {conv}} (S_{l}\, \lambda )\subseteq \operatorname {\mathrm {conv}} (S_{l}\, \mu )$ if and only if $\lambda _{+}\leq \mu _{+}$ .

Each orbit $S_{l}\, \lambda _{+}$ has a partial ordering induced by the Bruhat ordering on $S_{l}$ . More explicitly, this ordering is the transitive closure of the relation $s_{i}\lambda>\lambda $ if $\langle \alpha _{i}^{\vee } ,\lambda \rangle>0$ . We extend this to a partial ordering on all of $X = {\mathbb Z} ^{l}$ by defining $\lambda \leq \mu $ if $\lambda _{+}<\mu _{+}$ , or if $\lambda _{+} = \mu _{+} $ and $\lambda \leq \mu $ in the Bruhat order on $S_{l}\, \lambda _{+}$ .

Suppose $\langle \alpha _{i}^{\vee } ,\lambda \rangle \geq 0$ . If $\langle \alpha _{i}^{\vee } ,\lambda \rangle = 0$ , that is, if $\lambda =s_{i}\lambda $ , then

(70) $$ \begin{align} T_{i}\, x^{\lambda } = q\, x^{\lambda }. \end{align} $$

Otherwise, if $\langle \alpha _{i}^{\vee } ,\lambda \rangle> 0$ , then

(71) $$ \begin{align} \begin{aligned} T_{i}\, x^{\lambda } & \equiv q\, x^{s_{i}\lambda } + (q-1)x^{\lambda},\\ T_{i}\, x^{s_{i}\lambda } & \equiv x^{\lambda } \end{aligned} \end{align} $$

modulo the space spanned by monomials $x^{\mu }$ for $\mu $ strictly between $\lambda $ and $s_{i}\lambda $ on the root string $\lambda +{\mathbb Z} \alpha _{i}$ . Note that $\mu <\lambda $ for these weights $\mu $ since they lie on orbits strictly inside $\operatorname {\mathrm {conv}} (S_{l}\, \lambda )$ . Furthermore, the set of all weights $\mu \leq s_{i} \lambda $ is $s_{i}$ -invariant and has convex intersection with every root string $\nu +{\mathbb Z} \alpha _{i}$ ; hence, the space ${\mathbf k} \, \{x^{\mu }\mid \mu \leq s_{i}\lambda \}$ is closed under $T_{i}$ . It follows that if $\langle \alpha _{i}^{\vee }, \lambda \rangle \ge 0$ , then $T_{i}$ applied to any Laurent polynomial of the form $x^{\lambda }+\sum _{\mu < \lambda }c_{\mu } x^{\mu }$ yields a result of the form

(72) $$ \begin{align} T_{i}\, \bigl( x^{\lambda}+\sum_{\mu < \lambda}c_{\mu} x^{\mu} \bigr) = q\, x^{s_{i}\lambda } + \sum_{\mu < s_i \lambda} d_{\mu} x^{\mu}\,. \end{align} $$

4.3 Nonsymmetric Hall–Littlewood polynomials

For each $\operatorname {\mathrm {GL}} _{l}$ weight $\lambda \in {\mathbb Z} ^{l}$ , we define the nonsymmetric Hall–Littlewood polynomial

(73) $$ \begin{align} E_{\lambda }(x;q) = E_{\lambda }(x_1, \ldots, x_l;q) = q^{-\ell(w)}T_{w} x^{\lambda _{+}}, \end{align} $$

where $w\in S_{l}$ is such that $\lambda =w(\lambda _{+})$ . If $\lambda $ has nontrivial stabilizer, then w is not unique, but it follows from equations (70)–(72) that $E_{\lambda }(x;q)$ is independent of the choice of w and has the monic and triangular form

(74) $$ \begin{align} E_{\lambda }(x;q) = x^{\lambda } +\sum _{\mu <\lambda } c_{\mu } x^{\mu }. \end{align} $$

See Table 1 for examples.

Table 1 Nonsymmetric Hall–Littlewood polynomials $E^{\sigma }_{{\mathbf a}}(x_{1},x_{2},x_{3};q)$ and $F^{\sigma }_{{\mathbf a}}(y_{1},y_{2},y_{3};q)$ for $l = 3$ , $\sigma = 1$ , and $|{\mathbf a}| \le 2$ .

For context, we remark that several distinct notions of ‘nonsymmetric Hall–Littlewood polynomial’ can be found in the literature. Our $E_{\lambda }$ (and $F_{\lambda } $ , below) coincide with specializations of nonsymmetric Macdonald polynomials considered by Ion in [Reference Ion22, Theorem 4.8]. The twisted variants $E_{\lambda }^{\sigma }$ below are specializations of the ‘permuted basement’ nonsymmetric Macdonald polynomials studied (for $\operatorname {\mathrm {GL}} _{l}$ ) by Alexandersson [Reference Alexandersson1] and Alexandersson and Sawhney [Reference Alexandersson and Sawhney2]. We also note that $E_{\lambda }(x;q^{-1})$ and $F_{\lambda }(x;q)$ have coefficients in ${\mathbb Z}[q]$ and specialize at $q=0$ to Demazure characters and Demazure atoms, respectively.

For any $\mu \in {\mathbb R} ^{l}$ , we define $\operatorname {\mathrm {Inv}} (\mu ) = \{(i<j)\mid \mu _{i}>\mu _{j} \}$ . In the case of a permutation, $\operatorname {\mathrm {Inv}} (\sigma )$ is then the usual inversion set of $\sigma =(\sigma (1),\ldots ,\sigma (l)) \in S_l$ .

Taking $\rho $ as in §2.3 and $\epsilon>0$ small, the notation $\operatorname {\mathrm {Inv}} (\lambda +\epsilon \rho )$ denotes the set of pairs $i<j$ such that $\lambda _{i}\geq \lambda _{j}$ .

Given $\sigma \in S_{l}$ , we define twisted nonsymmetric Hall–Littlewood polynomials

(75) $$ \begin{align} E^{\sigma }_{\lambda }(x;q) = q^{\left|\operatorname{\mathrm{Inv}} (\sigma ^{-1})\cap \operatorname{\mathrm{Inv}} (\lambda +\epsilon \rho )\right|} T_{\sigma ^{-1}}^{-1} E_{\sigma ^{-1}(\lambda )}(x;q) \end{align} $$

(76) $$ \begin{align} F^{\sigma }_{\lambda }(x;q) = \overline{E^{\sigma w_{0}}_{-\lambda }(x;q)} = E^{\sigma w_{0}}_{-\lambda }(x_{1}^{-1},\ldots,x_{l}^{-1};q^{-1}), \end{align} $$

where $w_{0}\in S_{l}$ is the longest element, given by $w_{0}(i) = l+1-i$ . The normalization in equation (75) implies the recurrence

(77) $$ \begin{align} E^{\sigma }_{\lambda } = \begin{cases} q^{-I(\lambda _{i}\leq \lambda _{i+1})}\, T_{i}\, E^{s_{i}\sigma }_{s_{i}\lambda }, & s_{i}\sigma>\sigma \\ q^{I(\lambda _{i}\geq \lambda _{i+1})}\, T_{i}^{-1}\, E^{s_{i}\sigma }_{s_{i}\lambda }, & s_{i}\sigma <\sigma, \end{cases} \end{align} $$

where $I(P)=1$ if P is true, $I(P)=0$ if P is false. Together with the initial condition $E^{\sigma }_{\lambda } = x^{\lambda }$ for all $\sigma $ if $\lambda =\lambda _{+}$ , this determines $E^{\sigma }_{\lambda }$ for all $\sigma $ and $\lambda $ .

Proposition 4.3.2. For every $\sigma \in S_{l}$ , the elements $E_{\lambda }^{\sigma }(x;q)$ and $\overline {F_{\lambda }^{\sigma }(x;q)}$ are dual bases of ${\mathbf k} [x_{1}^{\pm 1},\ldots ,x_{l}^{\pm 1}]$ with respect to the inner product defined by

(78) $$ \begin{align} \langle f,g \rangle_{q} = \langle x^{0} \rangle\, f\, g\, \prod _{i<j} \frac{1-x_{i}/x_{j}}{1 - q^{-1} x_{i}/x_{j}}. \end{align} $$

In other words, $\langle E_{\lambda }^{\sigma }, \overline {F_{\mu }^{\sigma }} \rangle _{q} = \delta _{\lambda \mu }$ for all $\lambda ,\mu \in {\mathbb Z} ^{l}$ and all $\sigma \in S_{l}$ .

To prove Proposition 4.3.2, we need the following lemma.

Proof. It’s the same to show that $T_{i}-q$ is self-adjoint. A bit of algebra gives

(79) $$ \begin{align} T_{i}-q =q\, \frac{1-q^{-1} x_{i}/x_{i+1}}{1-x_{i}/x_{i+1}}(s_{i}-1), \end{align} $$

and therefore

(80) $$ \begin{align} \langle (T_{i}-q)f,g \rangle_{q} = q\, \langle x^{0} \rangle\, (s_{i}(f)\, g-f\, g) \prod \frac{1-x_{j}/x_{k}}{1-q^{-1}x_{j}/x_{k}}, \end{align} $$

where the product is over $j<k$ with $(j,k) \not = (i,i+1)$ . We want to show that this is symmetric in f and g, that is, that the right-hand side is unchanged if we replace $s_{i}(f)\, g$ with $f\, s_{i}(g)$ . Let $\Delta $ denote the product factor in equation (80), and note that $\Delta $ is symmetric in $x_{i}$ and $x_{i+1}$ . The constant term $\langle x^{0} \rangle \, \varphi (x)$ of any $\varphi (x_{1}, \ldots , x_{l})$ is equal to $\langle x^{0} \rangle \, s_{i} (\varphi (x))$ . In particular, $\langle x^{0} \rangle s_{i}(f) g\, \Delta = \langle x^{0} \rangle f s_{i}(g)\, \Delta $ , which implies the desired result.

Proof of Proposition 4.3.2.

The desired identity is just a tidy notation for $\langle E_{\lambda }^{\sigma },E_{-\mu }^{\sigma w_{0}} \rangle _{q} = \delta _{\lambda \mu }$ .

By equation (77), for every i, we have either $\langle E_{\lambda }^{\sigma }, E_{-\mu }^{\sigma w_{0}} \rangle _{q} = q^{e}\, \langle T_{i} E_{s_{i}\lambda }^{s_{i}\sigma }, T_{i}^{-1}E_{-s_{i} \mu }^{s_{i} \sigma w_{0}} \rangle _{q}$ or $\langle E_{\lambda }^{\sigma }, E_{-\mu }^{\sigma w_{0}} \rangle _{q} = q^{e}\, \langle T_{i}^{-1} E_{s_{i} \lambda }^{s_{i} \sigma }, T_{i} E_{- s_{i} \mu }^{s_{i} \sigma w_{0}} \rangle _{q}$ , depending on whether $s_{i}\sigma>\sigma $ or $s_{i}\sigma <\sigma $ , for some exponent e. Moreover, if $\lambda =\mu $ , then $q^{e} = 1$ .

Since $T_{i}$ is self-adjoint, we get $\langle E_{\lambda }^{\sigma }, E_{-\mu }^{\sigma w_{0}} \rangle _{q} = q^{e} \langle E_{s_{i}\lambda }^{s_{i}\sigma }, E_{-s_{i} \mu }^{s_{i} \sigma w_{0}} \rangle _{q}$ in either case. Repeating this gives an identity

(81) $$ \begin{align} \langle E^{\sigma }_{\lambda }, E^{\sigma w_{0}}_{-\mu } \rangle _{q} = q^{e} \langle E^{v \sigma }_{v \lambda }, E^{v \sigma w_{0}}_{-v \mu } \rangle _{q} \end{align} $$

for all $\lambda ,\mu \in {\mathbb Z} ^{l}$ and all $\sigma ,v\in S_{l}$ , again with $q^{e} = 1$ if $\lambda =\mu $ .

Choose $v\in S_{l}$ such that $\mu _{-} = v (\mu )$ is antidominant. Then equation (81) gives

(82) $$ \begin{align} \langle E^{\sigma }_{\lambda }, E^{\sigma w_{0}}_{-\mu } \rangle _{q} = q^{e} \langle E^{v \sigma }_{v \lambda }, x^{-(\mu _{-})} \rangle _{q} =q^{e}\, \langle x^{\mu _{-}} \rangle \, \Delta\, E^{v \sigma }_{v \lambda }, \end{align} $$

where now $\Delta $ is the product factor in equation (78). Let $\operatorname {\mathrm {supp}} (f)$ denote the set of weights $\nu $ for which $x^{\nu }$ occurs with nonzero coefficient in f. Since $\operatorname {\mathrm {supp}} (\Delta ) = Q_{+}$ , and $\operatorname {\mathrm {supp}} (E^{v \sigma }_{v \lambda })\subseteq \operatorname {\mathrm {conv}} (S_{l}\, \lambda )$ , it follows from equation (82) that if $\langle E^{\sigma }_{\lambda }, E^{\sigma w_{0}}_{-\mu } \rangle _{q}\not =0$ , then $(\mu _{-}-Q_{+})\cap \operatorname {\mathrm {conv}} (S_{l}\, \lambda )\not =\emptyset $ and therefore $\mu _{-} - \lambda _{-}\in Q_{+}$ . Since $w_{0}(Q_{+}) = -Q_{+}$ , this is equivalent to $\lambda _{+}\geq \mu _{+}$ .

By symmetry, exchanging $\lambda $ with $-\mu $ and $\sigma $ with $\sigma w_{0}$ , if $\langle E^{\sigma }_{\lambda }, E^{\sigma w_{0}}_{-\mu } \rangle _{q}\not =0$ , then we also have $(-\lambda )_{-} - (-\mu )_{-}\in Q_{+}$ , hence $\lambda _{+}-\mu _{+}\in -Q_{+}$ , that is, $\lambda _{+}\leq \mu _{+}$ . Hence, $\langle E^{\sigma }_{\lambda }, E^{\sigma w_{0}}_{-\mu } \rangle _{q}\not =0$ implies $\lambda _{+} = \mu _{+}$ , so $\lambda $ and $\mu $ belong to the same $S_{l}$ orbit. This reduces the problem to the case that $S_{l}\, \lambda = S_{l}\, \mu $ .

In this case, $(\mu _{-}-Q_{+})\cap \operatorname {\mathrm {conv}} (S_{l}\, \lambda ) = \{\mu _{-} \}$ . Furthermore, if $\lambda \not =\mu $ , then $v \lambda \not =\mu _{-}$ , and Corollary 4.3.1 implies that $(\mu _{-}-Q_{+})\cap \operatorname {\mathrm {supp}} (E^{v \sigma }_{v \lambda }) = \emptyset $ , hence $\langle E^{\sigma }_{\lambda }, E^{\sigma w_{0}}_{-\mu } \rangle _{q} =0$ .

If $\lambda =\mu $ , then the right-hand side of equation (82) reduces to $\langle x^{\mu _{-}} \rangle \, \Delta \, E^{v \sigma }_{\mu _{-}}$ . Since $\operatorname {\mathrm {supp}} (\Delta )=Q_{+}$ and $\operatorname {\mathrm {supp}} (E^{v \sigma }_{\mu _{-}})\subset \mu _{-}+Q_{+}$ , only the constant term of $\Delta $ and the $x^{\mu _{-}}$ term of $E^{v\sigma }_{\mu _{-}}$ contribute to the coefficient of $x^{\mu _{-}}$ in $\Delta \, E^{v \sigma }_{\mu _{-}}$ , and we have $\langle x^{\mu _{-}} \rangle E^{v\sigma }_{\mu _{-}} = 1$ by Corollary 4.3.1. Hence, $\langle E^{\sigma }_{\lambda }, E^{\sigma w_{0}}_{-\lambda } \rangle _{q} =1$ .

Lemma 4.3.4. Given $\lambda \in {\mathbb Z} ^{l}$ , suppose there is an index k such that $\lambda _{i}\geq \lambda _{j}$ for all $i\leq k$ and $j>k$ . Given $\sigma \in S_{l}$ , let $\sigma _{1}\in S_{k}$ and $\sigma _{2}\in S_{l-k}$ be the permutations such that $\sigma _{1}(1),\ldots ,\sigma _{1}(k)$ are in the same relative order as $\sigma (1),\ldots ,\sigma (k)$ , and $\sigma _{2}(1),\ldots ,\sigma _{2}(l-k)$ are in the same relative order as $\sigma (k+1),\ldots ,\sigma (l)$ . Then

(83) $$ \begin{align} E^{\sigma ^{-1}}_{\lambda }(x_{1},\ldots,x_{l};q) = E^{\sigma_{1} ^{-1}}_{(\lambda_{1},\ldots,\lambda _{k}) }(x_{1},\ldots,x_{k};q)\, E^{\sigma_{2} ^{-1}}_{(\lambda_{k+1},\ldots,\lambda _{l}) }(x_{k+1},\ldots,x_{l};q). \end{align} $$

4.4 LLT series

Definition 4.4.1. Given $\operatorname {\mathrm {GL}} _{l}$ weights $\alpha ,\beta \in {\mathbb Z} ^{l}$ and a permutation $\sigma \in S_{l}$ , the LLT series ${\mathcal L}^{\sigma } _{\beta /\alpha }(x_{1},\ldots ,x_{l};q)$ is the infinite formal sum of irreducible $\operatorname {\mathrm {GL}} _{l}$ characters in which the coefficient of $\chi _{\lambda }$ is defined by

(84) $$ \begin{align} \langle \chi _{\lambda } \rangle\, {\mathcal L} ^{\sigma ^{-1}}_{\beta /\alpha }(x;q^{-1}) = \langle E^{\sigma }_{\beta } \rangle \, \chi _{\lambda }\, E^{\sigma }_{\alpha }, \end{align} $$

where $E^{\sigma }_{\lambda }(x;q)$ are the twisted nonsymmetric Hall–Littlewood polynomials from §4.3.

We remark that the coefficients of $E^{\sigma }_{\lambda }(x;q)$ are polynomials in $q^{-1}$ , so the convention of inverting q in equation (84) makes the coefficients of ${\mathcal L} ^{\sigma }_{\beta /\alpha }(x;q)$ polynomials in q. Inverting $\sigma $ as well leads to a more natural statement and proof in Corollary 4.5.7, below.

Proposition 4.4.2. We have the formula

(85) $$ \begin{align} {\mathcal L} ^{\sigma }_{\beta /\alpha }(x;q) = {\mathbf H} _{q}(w_{0}(F^{\sigma ^{-1} }_{\beta }(x;q) \overline{E^{\sigma ^{-1}}_{\alpha }(x;q)})), \end{align} $$

where ${\mathbf H} _{q}$ is the Hall–Littlewood symmetrization operator in equation (23) and $w_{0}(i)=l+1-i$ is the longest element in $S_{l}$ .

Proof. By Proposition 4.3.2, the coefficient $ \langle E^{\sigma ^{-1}}_{\beta }(x;q^{-1}) \rangle \, \chi _{\lambda } \, E^{\sigma ^{-1}}_{\alpha }(x;q^{-1})$ of $\chi _{\lambda }$ in ${\mathcal L} ^{\sigma }_{\beta /\alpha }$ is given by the constant term

(86) $$ \begin{align} \langle x^{0} \rangle\, \chi _{\lambda }\, F^{\sigma ^{-1}}_{\beta }(x^{-1};q)\, E^{\sigma ^{-1}}_{\alpha }(x;q^{-1}) \prod _{i<j}\frac{1-x_{i}/x_{j}}{1-q\, x_{i}/x_{j}}. \end{align} $$

Substituting $x_{i}\mapsto x_{i}^{-1}$ and applying $w_{0}$ , this is equal to

(87) $$ \begin{align} \langle x^{0} \rangle\, \overline{\chi _{\lambda }}\, w_{0}(F^{\sigma ^{-1}}_{\beta }(x;q) \overline{E^{\sigma ^{-1}}_{\alpha }(x;q)}) \prod _{i<j}\frac{1-x_{i}/x_{j}}{1-q\, x_{i}/x_{j}}. \end{align} $$

Considering this expression as a formal Laurent series in q and applying equation (20) coefficient-wise yields

(88) $$ \begin{align} \langle \chi _{\lambda } \rangle {\boldsymbol \sigma } \left( w_{0}(F^{\sigma^{-1}}_{\beta }(x;q) \overline{E^{\sigma ^{-1}}_{\alpha }(x;q)}) \prod_{i<j}\frac{1}{1-q\, x_{i}/x_{j}} \right), \end{align} $$

which is $\langle \chi _{\lambda } \rangle {\mathbf H} _{q}(w_{0}(F^{\sigma ^{-1} }_{\beta }(x;q) \overline {E^{\sigma ^{-1}}_{\alpha }(x;q)}))$ , as desired.

4.5 Tableaux for LLT series

We now work out a tableau formalism which relates the polynomial part ${\mathcal L} ^{\sigma }_{\beta /\alpha }(x;q)_{\operatorname {\mathrm {pol}} }$ to a combinatorial LLT polynomial ${\mathcal G} _{\nu }(x;q)$ .

Lemma 4.5.1. For all $\sigma \in S_{l}$ , $\lambda \in {\mathbb Z} ^{l}$ and k, the product of the elementary symmetric function $e_{k}(x)$ and the nonsymmetric Hall–Littlewood polynomial $E^{\sigma ^{-1} }_{\lambda }(x;q)$ is given by

(89) $$ \begin{align} e_{k}(x)\, E^{\sigma ^{-1}}_{\lambda }(x;q) = \sum _{|I| = k} q^{-h_{I}} E^{\sigma ^{-1}}_{\lambda + \varepsilon _{I}}(x;q), \end{align} $$

where $I\subseteq \{1,\ldots ,l \}$ has k elements, $\varepsilon _{I} = \sum _{i\in I}\varepsilon _{i}$ is the indicator vector of I, and

(90) $$ \begin{align} h_{I} = \left|\, \operatorname{\mathrm{Inv}} (\lambda + \varepsilon _{I} + \epsilon \sigma )\setminus \operatorname{\mathrm{Inv}} (\lambda + \epsilon \sigma )\, \right|. \end{align} $$

Equivalently, $h_{I}$ is the number of pairs $i<j$ such that $i\in I$ , $j\not \in I$ , and we have $\lambda _{j} = \lambda _{i}$ if $\sigma (i)<\sigma (j)$ , or $\lambda _{j} = \lambda _{i}+1$ if $\sigma (i)>\sigma (j)$ .

Proof. First, consider the case $\sigma =1$ . Being symmetric, $e_{k}(x)$ commutes with $T_{w}$ , giving

(91) $$ \begin{align} e_{k}\, E_{\lambda } = q^{-\ell(w)} T_{w}\, e_{k}\, x^{\lambda _{+}} = q^{-\ell(w)}\sum _{|J| = k} T_{w}\, x^{\lambda _{+}+\varepsilon _{J}}, \end{align} $$

where $\lambda = w(\lambda _{+})$ , as in equation (73). To fix the choice, we take w maximal in its coset $w\, \operatorname {\mathrm {Stab}} (\lambda _{+})$ .

In each term of the sum in equation (91), the weight $\mu =\lambda _{+}+\varepsilon _{J}$ can fail to be dominant at worst by having some entries $\mu _{j} = \mu _{i}+1$ for indices $i<j$ such that $(\lambda _{+})_{i} = (\lambda _{+})_{j}$ , $i\not \in J$ and $j\in J$ . Let v be the minimal permutation such that $\mu _{+} = v(\mu )$ , that is, the permutation that moves indices $j\in J$ to the left within each block of constant entries in $\lambda _{+}$ . The formula $T_{i} x_{i}^{a}x_{i+1}^{a+1} = x_{i}^{a+1}x_{i+1}^{a}$ is immediate from the definition of $T_{i}$ and implies that $T_{v} x^{\mu } = x^{\mu _{+}}$ . By the maximality of w, since $v\in \operatorname {\mathrm {Stab}} (\lambda _{+})$ , there is a reduced factorization $w = uv$ , hence $T_{w} = T_{u}T_{v}$ . Then

(92) $$ \begin{align} T_{w}\, x^{\lambda _{+}+\varepsilon _{J}} = T_{u} x^{\mu _{+}} = q^{\ell(u)} E_{\lambda\ {+}\ w(\varepsilon _{J})} \end{align} $$

since $\lambda +w(\varepsilon _{J}) = w(\mu ) = u(\mu _{+})$ .

Now, $\ell (v)$ is equal to the number of pairs $i<j$ such that $\mu _{i}<\mu _{j}$ , that is, such that $(\lambda _{+})_{i}= (\lambda _{+})_{j}$ , $i\not \in J$ and $j\in J$ . By maximality, the permutation w carries these to the pairs $j' = w(i)$ , $i' = w(j)$ such that $i' < j'$ , $\lambda _{i'} = \lambda _{j'}$ , $i'\in I$ and $j'\not \in I$ , where $I = w(J)$ . For $\sigma =1$ , the definition of $h_{I}$ is the number of such pairs $i',j'$ , so we have $\ell (u)-\ell (w) = -\ell (v) = -h_{I}$ . Hence, the term for J in equation (91) is $q^{-\ell (w)} T_{w} x^{\lambda _{+}+\varepsilon _{J}} = q^{-h_{I}} E_{\lambda +\varepsilon _{I}}$ . As J ranges over subsets of size k, so does $I = w(J)$ , giving equation (89) in this case.

Substituting $\sigma (\lambda )$ for $\lambda $ and $\sigma (I)$ for I in the $\sigma =1$ case and acting on both sides with $T_{\sigma }^{-1}$ yields

(93) $$ \begin{align} q^{-\left|\operatorname{\mathrm{Inv}} (\sigma )\cap \operatorname{\mathrm{Inv}} (\lambda +\epsilon \rho )\right|} e_{k}\, E^{\sigma ^{-1}}_{\lambda } = \sum _{|I|=k} q^{-|\operatorname{\mathrm{Inv}} (\sigma (\lambda +\varepsilon _{I}))\setminus \operatorname{\mathrm{Inv}} (\sigma (\lambda ))|} q^{-\left|\operatorname{\mathrm{Inv}} (\sigma )\cap \operatorname{\mathrm{Inv}} (\lambda +\varepsilon _{I} + \epsilon \rho )\right|} E^{\sigma ^{-1}}_{\lambda +\varepsilon _{I}}. \end{align} $$

Combining powers of q gives the desired identity (89) if we verify that

(94) $$ \begin{align} &\left|\, \operatorname{\mathrm{Inv}} (\sigma )\cap \operatorname{\mathrm{Inv}} (\lambda +\varepsilon _{I} + \epsilon \rho )\, \right| - \left|\, \operatorname{\mathrm{Inv}} (\sigma )\cap \operatorname{\mathrm{Inv}} (\lambda +\epsilon \rho )\, \right| \notag\\ &\qquad\qquad\qquad\qquad = \left|\, \operatorname{\mathrm{Inv}} (\lambda + \varepsilon _{I} + \epsilon \sigma )\setminus \operatorname{\mathrm{Inv}} (\lambda + \epsilon \sigma )\, \right| - \left|\, \operatorname{\mathrm{Inv}} (\sigma (\lambda +\varepsilon _{I}))\setminus \operatorname{\mathrm{Inv}} (\sigma (\lambda ))\, \right|. \end{align} $$

On the left-hand side, cancelling the contribution from the intersection of the two sets leaves

(95) $$ \begin{align} \left| \operatorname{\mathrm{Inv}} (\sigma )\cap (\operatorname{\mathrm{Inv}} (\lambda +\varepsilon _{I} + \epsilon \rho )\setminus \operatorname{\mathrm{Inv}} (\lambda + \epsilon \rho )) \right| - \left| \operatorname{\mathrm{Inv}} (\sigma )\cap (\operatorname{\mathrm{Inv}} (\lambda +\epsilon \rho )\setminus \operatorname{\mathrm{Inv}} (\lambda +\varepsilon _{I} + \epsilon \rho )) \right|. \end{align} $$

The first term in formula (95) counts pairs $i<j$ such that $i\in I$ , $j\not \in I$ , $\sigma (i)>\sigma (j)$ , and $\lambda _{j} = \lambda _{i}+1$ . The second term counts pairs $i>j$ such that $i\in I$ , $j\not \in I$ , $\sigma (i)<\sigma (j)$ , and $\lambda _{j} = \lambda _{i}$ . The first term on the right-hand side of equation (94) counts pairs $i<j$ such that $i\in I$ , $j\not \in I$ , and $\lambda _{j} = \lambda _{i}$ if $\sigma (i)<\sigma (j)$ , or $\lambda _{j} = \lambda _{i}+1$ if $\sigma (i)>\sigma (j)$ . The second term on the right-hand side of equation (94) counts the set of pairs whose images under $\sigma ^{-1}$ are pairs $i,j$ (in either order) such that $i\in I$ , $j\not \in I$ , $\sigma (i)<\sigma (j)$ and $\lambda _{i} = \lambda _{j}$ . The cases in the first term with $\sigma (i)<\sigma (j)$ cancel those in the second term with $i<j$ . The remaining cases in the first term, with $\sigma (i)>\sigma (j)$ , match the first term in formula (95), while the remaining cases in the second term, with $i>j$ , match the second term in formula (95). This proves equation (94) and completes the proof of the lemma.

Given $\alpha ,\beta \in {\mathbb Z} ^{l}$ such that $\alpha _{j}\leq \beta _{j}$ for all j, we let $\beta /\alpha $ denote the tuple of one-row skew shapes $(\beta _{j})/(\alpha _{j})$ such that the x coordinates of the right edges of boxes in the row that forms the j-th component are the integers $\alpha _{j}+1,\ldots ,\beta _{j}$ . The boxes just outside this j-th component, adjacent to the left and right ends of the row, then have right edges with x coordinate $\alpha _{j}$ and $\beta _{j}+1$ . In the case of an empty row with $\alpha _{j} = \beta _{j}$ , we still consider these two boxes to be adjacent to the ends of the row.

For each box a, let $i(a)$ denote the x coordinate of its right edge and $j(a)$ the index of the row containing it.

For $\sigma \in S_{l}$ , we define $\sigma (\beta /\alpha ) = \sigma (\beta )/\sigma (\alpha )$ . If a is a box in row $j(a)$ of $\beta /\alpha $ , then $\sigma (a)$ denotes the corresponding box with x coordinate $i(\sigma (a)) = i(a)$ in row $\sigma (j(a))$ of $\sigma (\beta /\alpha )$ . The adjusted content of $\sigma (a)$ , as defined in §4.1, is then $\tilde {c} (\sigma (a)) = i(a)+ \sigma (j(a))\, \epsilon $ . Hence, the reading order on $\sigma (\beta /\alpha )$ corresponds via $\sigma $ to the ordering of boxes in $\beta /\alpha $ by increasing values of $i(a) + \sigma (j(a))\, \epsilon $ .

We define a $\sigma $ -triple in $\beta /\alpha $ to consist of any three boxes $(a,b,c)$ arranged as follows: Boxes a and c are in or adjacent to the same row $j(a) = j(c)$ and are consecutive, that is, $i(c) = i(a)+1$ , while box b is in a row $j(b)<j(a)$ , and we have $a<b<c$ in the ordering corresponding to the reading order on $\sigma (\beta /\alpha )$ . More explicitly, this means that, if $\sigma (j(b))<\sigma (j(a))$ , then $i(b) = i(c)$ , while if $\sigma (j(b))>\sigma (j(a))$ , then $i(b) = i(a)$ . The box b is required to be a box of $\beta /\alpha $ , but box a is allowed to be outside and adjacent to the left end of a row, while c is similarly allowed to be outside and adjacent to the right end of a row.

An example of a tuple $\beta /\alpha $ , with all its $\sigma $ -triples for two different choices of $\sigma $ , is shown in Figure 1.

A negative tableau on $\beta /\alpha $ is a map $S\colon \beta /\alpha \rightarrow {\mathbb Z} _{>0} $ strictly increasing on each row. In the terminology of §4.1, S is a super tableau on $\beta /\alpha $ with entries in ${\mathbb Z} _{>0} $ , considered as a negative alphabet ordered by $\overline {1}<\overline {2}<\cdots $ . We say that a $\sigma $ -triple $(a,b,c)$ in $\beta /\alpha $ is increasing in S if $S(a)<S(b)<S(c)$ , with the convention that $S(a) = -\infty $ if a is just outside the left end of a row and $S(c) = \infty $ if c is just outside right end of a row. Along with the $\sigma $ -triples in $\beta /\alpha $ , Figure 1 also displays which triples are increasing in an example tableau S.

Proposition 4.5.2. Given $\alpha ,\beta \in {\mathbb Z} ^{l}$ such that $\alpha _{j}\leq \beta _{j}$ for all j, and $\sigma \in S_{l}$ , let

(96) $$ \begin{align} N^{\sigma }_{\beta /\alpha }(X;q) = \sum_{S\in\operatorname{\mathrm{SSYT}}_{-}(\beta/\alpha)} q^{h_{\sigma}(S)} x^S \end{align} $$

be the generating function for negative tableaux S on the tuple of one-row skew diagrams $(\beta _{j})/(\alpha _{j})$ , weighted by $q^{h_{\sigma }(S)}$ , where $h_{\sigma }(S)$ is the number of increasing $\sigma $ -triples in S. Then $N^{\sigma }_{\beta /\alpha }(X;q)$ is a symmetric function, and $\omega N^{\sigma }_{\beta /\alpha }(X;q)$ evaluates in l variables to

(97) $$ \begin{align} (\omega N^{\sigma }_{\beta /\alpha })(x_{1},\ldots,x_{l};q) = {\mathcal L} ^{\sigma }_{\beta /\alpha }(x_{1},\ldots,x_{l};q)_{\operatorname{\mathrm{pol}} }. \end{align} $$

If we do not have $\alpha _{j}\leq \beta _{j}$ for all j, then ${\mathcal L} ^{\sigma }_{\beta /\alpha }(x;q)_{\operatorname {\mathrm {pol}} } = 0$ .

Proof. Let $L^{\sigma }_{\beta /\alpha }(X;q)$ be the unique symmetric function such that (i) $L^{\sigma }_{\beta /\alpha }(X;q)$ is a linear combination of Schur functions $s_{\lambda }$ with $\ell (\lambda )\leq l$ , and (ii) in l variables, it evaluates to

(98) $$ \begin{align} L^{\sigma }_{\beta /\alpha }(x_{1},\ldots,x_{l};q) = {\mathcal L} ^{\sigma }_{\beta /\alpha }(x_{1},\ldots,x_{l};q)_{\operatorname{\mathrm{pol}} }. \end{align} $$

What we need to prove is that $\omega \, L^{\sigma }_{\beta /\alpha }(X;q) = N^{\sigma }_{\beta /\alpha }(X;q)$ .

The definition of ${\mathcal L} ^{\sigma }_{\beta /\alpha }(x;q)$ implies that $L^{\sigma }_{\beta /\alpha }(X;q)$ satisfies

(99) $$ \begin{align} \langle s_{\lambda }(X), L^{\sigma }_{\beta /\alpha }(X;q) \rangle = \langle E^{\sigma ^{-1}}_{\beta }(x;q^{-1}) \rangle\, s_{\lambda }(x_{1},\ldots,x_{l})\, E^{\sigma ^{-1}}_{\alpha }(x;q^{-1}) \end{align} $$

for every partition $\lambda $ , including when $\ell (\lambda )> l$ since then both sides are zero. By linearity, we can replace $s_{\lambda }$ by any symmetric function f, giving

(100) $$ \begin{align} \langle f(X),\, L^{\sigma}_{\beta/\alpha}(X;q) \rangle = \langle E^{\sigma ^{-1}}_{\beta }(x;q^{-1}) \rangle\, f(x)\, E^{\sigma ^{-1}}_{\alpha }(x;q^{-1})\,. \end{align} $$

The coefficient of $m_{\mu }(X)$ in $\omega \, L^{\sigma }_{\beta /\alpha }(X;q)$ is given by taking $f=e_{\mu }$ .

To show that $\omega \, L^{\sigma }_{\beta /\alpha }(X;q)$ is given by the tableau generating function in equation (96), we use Lemma 4.5.1 to express

(101) $$ \begin{align} \langle E^{\sigma ^{-1}}_{\beta }(x;q^{-1}) \rangle\, e_{\mu }(x)\, E^{\sigma ^{-1}}_{\alpha }(x;q^{-1}) \end{align} $$

as a sum of powers of q indexed by negative tableaux. In particular, this coefficient will vanish unless we have $\alpha _{j}\leq \beta _{j}$ for all j, giving the last conclusion in the proposition.

Multiplying by $e_{\mu _{1}}$ through $e_{\mu _{n}}$ successively and keeping track of one chosen term in each product gives a sequence of terms $E^{\sigma ^{-1}}_{\alpha ^{(0)}}, E^{\sigma ^{-1}}_{\alpha ^{(1)}},\ldots , E^{\sigma ^{-1}}_{\alpha ^{(n)}}$ , in which $\alpha ^{(0)} = \alpha $ and $\alpha ^{(m+1)} = \alpha ^{(m)} + \varepsilon _{I}$ for a set of indices I of size $\mu _{m}$ , for each m. Each sequence with $\alpha ^{(n)} = \beta $ contributes to formula (101).

If we record these data in the form of a tableau $S\colon \beta /\alpha \rightarrow {\mathbb Z} _{>0} $ with $S(a) = m$ for $a\in (\alpha ^{(m)}/\alpha ^{(m-1)})$ , S satisfies the condition that it is a negative tableau of weight $x^{S} = x^{\mu }$ . The contribution to formula (101) from the corresponding sequence of terms is the product of the $q^{h_{I}}$ with $h_{I}$ as in equation (90) for $k= \mu _{m}$ , $\lambda =\alpha ^{(m-1)}$ and I the set of indices j such that $S(a) = m$ for some box a in row j.

We now express the $h_{I}$ corresponding to $(\alpha ^{(m)}/\alpha ^{(m-1)}) = S^{-1}(\{m \})$ as an attribute of S. For $h_{I}$ to count a pair $i<j$ , we must have $i\in I$ , which means that $S(b) = m$ for a box b in row i, and $j\not \in I$ , and one of the following two situations.

If $\sigma (i)<\sigma (j)$ , we must have $\alpha ^{(m-1)}_{j} = \alpha ^{(m-1)}_{i}$ . Since there is no m in row j of S, this means that the boxes a and c in row j with coordinates $i(a) = i(b)-1$ , $i(c) = i(b)$ have $S(a)<S(b)<S(c)$ , with the same convention as above that $S(a) = -\infty $ if a is to the left of a row of $\beta /\alpha $ , and $S(c) = \infty $ if c is to the right of a row.

If $\sigma (i)>\sigma (j)$ , we must have $\alpha ^{(m-1)}_{j} = \alpha ^{(m-1)}_{i}+1$ . This means that the boxes a and c in row j with coordinates $i(a) = i(b)$ , $i(c) = i(b)+1$ have $S(a)<S(b)<S(c)$ , with the same convention as before.

These two cases establish that $h_{I}$ is equal to the number of increasing $\sigma $ -triples in S for which $S(b) = m$ . Summing them up gives the total number of increasing $\sigma $ -triples, implying that the coefficient in formula (101) is the sum of $q^{h_{\sigma }(S)}$ over negative tableaux S of weight $x^{S} = x^{\mu }$ on $\beta /\alpha $ , where $h_{\sigma }(S)$ is the number of increasing $\sigma $ -triples in S.

Similar to the notation $h_{\sigma }(S)$ in equation (96) for the number of increasing triples in a tableau S, we let $h_{\sigma }(\beta /\alpha )$ denote the total number of triples in a tuple of rows.

Lemma 4.5.3. Given $\sigma \in S_l$ and $\alpha ,\beta \in {\mathbb Z} ^{l}$ with $\alpha _{j}\leq \beta _{j}$ for all j, we have

(102) $$ \begin{align} h_{\sigma }(\beta /\alpha )-\operatorname{\mathrm{inv}} (T)= h_{\sigma }(S) \end{align} $$

for $T \in \operatorname {\mathrm {SSYT}}_{-}(\sigma (\beta /\alpha ))$ , where $S = \sigma ^{-1}(T) = T\circ \sigma $ .

Example 4.5.4. Let S be as in Figure 1 and

so T is a negative tableau on $\sigma (\beta /\alpha ) = (3,5,6)/(1,2,2)$ . Reading T by reading order on $\sigma (\beta /\alpha )$ gives $1\, 3\, 4\, 2\, 7\, 4\, 9\, 6\, 7$ . The pairs $(T(a),T(b))$ for attacking inversions $(a,b)$ in T are $(3,2)$ , $(4,2)$ , $(7,4)$ and $(9,6)$ , so $\operatorname {\mathrm {inv}}(T) = 4$ . From the last diagram in Figure 1, we have $h_{\sigma }(\beta /\alpha ) = 5$ , $h_{\sigma }(S)= 1$ , so $h_{\sigma }(\beta /\alpha ) - \operatorname {\mathrm {inv}}(T) = h_{\sigma }(S)$ is indeed satisfied.

Now, let $\sigma = 1$ and $T = S \circ \sigma ^{-1} = S$ . Reading T by reading order on $\beta /\alpha $ gives $1\, 2\, 3\, 4\, 4\, 7\, 6\, 9\, 7$ . The pairs $(T(a),T(b))$ for the attacking inversions are $(4,4)$ , $(7,6)$ and $(9,7)$ , so $\operatorname {\mathrm {inv}}(T)= 3$ ; note that equal attacking entries $(4,4)$ count as an inversion in a negative tableau. From the second diagram in Figure 1, we see $h_{\sigma }(\beta /\alpha ) = 7$ , $h_{\sigma }(S)= 4$ , so again $h_{\sigma }(\beta /\alpha ) - \operatorname {\mathrm {inv}}(T) = h_{\sigma }(S)$ holds.

The relation between ${\mathcal G}$ and ${\mathcal L}$ can now be made precise by applying Proposition 4.5.2 and Lemma 4.5.3 to the expression for $q^{h_{\sigma }(\beta /\alpha )} {\mathcal G} _{\sigma (\beta /\alpha )}(X;q^{-1})$ given in Corollary 4.1.4.

Corollary 4.5.7. Given $\alpha ,\beta \in {\mathbb Z} ^{l}$ and $\sigma \in S_{l}$ ,

(103) $$ \begin{align} {\mathcal L} ^{\sigma }_{\beta /\alpha }(x;q)_{\operatorname{\mathrm{pol}} } = \begin{cases} q^{h_{\sigma }(\beta/\alpha )} {\mathcal G} _{\sigma (\beta /\alpha )}(x;q^{-1}) & \text{if } \alpha _{j}\leq \beta _{j} \text{ for all } j\\ 0 & \text{otherwise}, \end{cases} \end{align} $$

where $h_{\sigma }(\beta /\alpha )$ is the number of $\sigma $ -triples in $\beta /\alpha $ , as in Lemma 4.5.3, and the right-hand side is evaluated in l variables $x_{1},\ldots , x_{l}$ .

5.1 Cauchy identity

In this section, we derive our main results, Theorems 5.3.1 and 5.5.1. The key point is the following delightful ‘Cauchy identity’ for nonsymmetric Hall–Littlewood polynomials.

Theorem 5.1.1. For any permutation $\sigma \in S_{l}$ , the twisted nonsymmetric Hall–Littlewood polynomials $E^{\sigma }_{\lambda }(x;q)$ and $F^{\sigma }_{\lambda }(x;q)$ in (75, 76) satisfy the identity

(104) $$ \begin{align} \frac{\prod _{i<j} (1 - q\, t\, x_{i} \, y_{j})}{\prod _{i\leq j} (1 - t\, x_{i}\, y_{j})} = \sum _{{\mathbf a} \geq 0} t^{|{\mathbf a} |}\, E^{\sigma }_{{\mathbf a} }(x_{1},\ldots,x_{l};q^{-1}) \, F^{\sigma }_{{\mathbf a} }(y_{1},\ldots,y_{l};q), \end{align} $$

where the sum is over ${\mathbf a} = (a_{1},\ldots ,a_{l})$ with all $a_{i}\geq 0$ and $|{\mathbf a} | \overset {\text {{def}}}{=} a_{1}+\cdots +a_{l}$ .