Signature characters of invariant Hermitian forms on irreducible Verma modules of singular infinitesimal character and Hall–Littlewood polynomials

Abstract The Unitary Dual Problem is one of the most important open problems in mathematics: classify the irreducible unitary representations of a given group. It is known for a real reductive Lie group that A q ( λ ) $A_{\mathfrak {q}}(\lambda )$ upper A Subscript German q Baseline left parenthesis lamda right parenthesis modules are unitary and that any unitarizable Harish-Chandra module of strongly regular infinitesimal character is isomorphic to an A q ( λ ) $A_{\mathfrak {q}}(\lambda )$ upper A Subscript German q Baseline left parenthesis lamda right parenthesis . Thus, it is of interest to study representations of singular infinitesimal character. For a compact real form and any alcove of the form w ( − λ + A ― ∘ ) , $w(-\lambda + \underline {A}_\circ ),$ w left parenthesis negative lamda plus upper A underbar Subscript ring Baseline right parenthesis comma where λ $\lambda $ lamda is dominant (possibly singular) and A ― ∘ $\underline {A}_\circ $ upper A underbar Subscript ring is the dominant fundamental alcove, the signature character of the canonical invariant Hermitian form on the irreducible Verma module of infinitesimal character in that alcove is the “negative” of a Hall–Littlewood polynomial summand at q = − 1 $q=-1$ q equals negative 1 times a version of the Weyl denominator. (Signature characters for other real forms and alcoves of other forms may also be expressed using Hall–Littlewood polynomial summands.) Such formulas give hope that the Unitary Dual Problem is tractable in the singular case.