---

APPEARED IN BULLETIN OF THE

arXiv:math/9301214v1 [math.RT] 1 Jan 1993APPEARED IN BULLETIN OF THEAMERICAN MATHEMATICAL SOCIETYVolume 28, Number 1, January 1993, Pages 1-74HOMOGENEOUS FUNCTIONS ON LIGHT CONES:THE INFINITESIMAL STRUCTURE OF SOME DEGENERATEPRINCIPAL SERIES REPRESENTATIONSROGER E. HOWE AND ENG-CHYE TAN1. IntroductionOne of the classic and most pleasing calculations in the representation theory ofsemisimple Lie groups is Bargmann’s [Ba] (see also [Na]) description of the structureof representations of SL(2, R).

Bargmann constructs a basis of eigenvectors for thecompact subgroup SO(2) ⊆SL(2, R) and then explicitly describes the action ofthe Lie algebra of SL(2, R) on this basis. From the straightforward and explicitformulas that result, it is easy to make conclusions about unitarity, irreducibility,isomorphism, and other natural questions concerning these representations.Unfortunately Bargmann’s example stands almost alone; the literature tendsto give the impression that once beyond the civilized confines of SL(2, R), oneimmediately finds oneself in a dense thicket of difficulties, where explicit calculationis difficult and of little value.

The purpose of this paper is to point to an infiniteclass of examples in which the pleasant characteristics of Bargmann’s work can beretained; one can describe a model for representations in which the action of theLie algebra is given by explicit simple formulas, which permit direct determinationof important structural properties of the representations.The examples we will treat are spaces of functions that live on light cones andsatisfy appropriate homogeneity conditions. These spaces are modules for an ap-propriate isometry group—O(p, q), U(p, q), or Sp(p, q).

In case q = 1, the sphericalprincipal series are among our examples, and Kostant’s celebrated results [Ko] onthe complementary series can be read offfrom our formulas, excluding the one ex-ceptional rank one group. (Essentially this calculation was done, in less systematicfashion, in [Jo, JW].

)We are hardly the first to study these representations. Significant parts of ourresults, including Dixmier’s [Di] study of the De Sitter group, Hirai’s notes onO(p, 1) and U(p, 1) [Hi1, Hi2], Kostant’s work [Ko], the calculations of Johnsonand Wallach [JW] for the spherical principal series, the studies of Schlichtkrull [Sc],Sekiguchi [Se], Strichartz [St1, St2], Rallis and Schiffmann [RS], Rossmann [Ro],Molcanov [Mo], Faraut [Fa]on functions on hyperboloids, the calculations of Klimyk and various colleagues(see [KG] and the references therein) and others [Br, Cl, Cw, FR, GN, HC1, JV,Received by the editors January 27, 1992.1991 Mathematics Subject Classification.

Primary 22E46; Secondary 17B10.Key words and phrases. Complementary series, composition series, degenerate principal seriesrepresentations, K-type diagrams, light cones, unitary representations.The first author was partially supported by NSF Grant DMS #9103608.c⃝1993 American Mathematical Society0273-0979/93 $1.00 + $.25 per page1

2R. E. HOWE AND E. C. TANNi, Sa, Sp1, Tk, Vi] are already in the literature.

The paper [Br] has diagramsthat resemble some of ours. The unitary duals of the rank one groups have beenclassified (see [BS, Cl, Hi1, Hi2] and the references therein).

In fact, most or all ofour results may be in the literature, at least implicitly. The main purpose of thepresent paper is to make our results explicit and to emphasize how concrete andhow simple a picture one may draw of these representations.

We use virtually notechnology. There is no clear reason why these calculations could not have beendone in the 1940s or early 1950s.

If they had been, the development of semisimplerepresentation theory might have been substantially different. For instance, besidesthe results of Kostant mentioned above, it is easy to read offfrom our results theunitary duals of O(2, 1), O(4, 1), and U(2, 1).

The last of these was not known untilthe mid 1960s [Hi2].Here is a summary of the paper.In §2 we set the context and describe ourrepresentations in the case of the indefinite orthogonal groups O(p, q). We describethe K-spectrum of the representations, where K = O(p) × O(q) is the maximalcompact subgroup of O(p, q), and we compute the infinitesimal action of the Liealgebra o(p, q) in terms of the K-spectrum.

On the basis of the formulas for theo(p, q) action, we determine composition series for our representations. It turnsout that the structure of the composition series involves some integrality conditionswhich make it depend in a significant way on the parities of p and q.

Also, we mustdistinguish the cases when p or q is less than or equal to 2, since these involve somedegeneration in the K-spectrum that affects the representation structure.In §3 we apply our formulas to determine which of our representations carryinvariant Hermitian forms, and on which subquotients these forms can be definite,that is, which constituents of our representations are unitary. In particular, wedetermine the extent of the complementary series and what happens at the end ofthe complementary series.In §§4 and 5 we adapt the discussion of §§2, 3 to the unitary groups U(p, q) andthe indefinite symplectic groups Sp(p, q).

The overall features of the picture forthese groups is controlled by O(p, q), as is made clear by Lemmas 4.1 and 5.3, butdifferences in the K-spectrum in the degenerate cases when q = 1 lead to interestingnew phenomena, notably considerable variation in the length of the complementaryseries, including Kostant’s results concerning the spherical complementary series ofU(p, 1) and Sp(p, 1).In order not to distract from the coherence and directness of the phenomenathat we present, we omit the details of various calculations in §§2–5. These areworked out in §6.

The final §7 discusses how our results fit into the general theoryof representations of semisimple Lie groups.To conclude this introduction, we note that those readers who are familiar withthe theory of reductive dual pairs [Ho3] will find our results can be profitablyinterpreted in terms of this theory, which can be used to predict much of whatwe present. We emphasize again, however, that our elementary and computationalapproach is independent of this or any general theory (except that we use [Ho3] asa convenient reference for facts about spherical harmonics that must somewhere bein the classical literature).

DEGENERATE PRINCIPAL SERIES32. The O(p, q)-module structure of homogeneous functionson light cones: K-types and composition seriesLet Rp+q ≃Rp ⊕Rq denote the vector space of tuples w = (x, y) wherex =x1x2...xp∈Rpandy =y1y2...yq∈Rq.To fix ideas, we will take p ≥q.

We consider Rp+q to be endowed with the indefiniteinner product(w, w′)p,q = (x, x′)p −(y, y′)q =pXj=1xjx′j −qXj=1yjy′j.We will use the notationr2p,q(w) = (w, w)p,qand likewise for r2p(x), r2q(y).Let O(p, q) denote the isometry group of (·, ·)p,q; that is, O(p, q) is the subgroupof linear transformations g ∈GL(p + q, R) such that(gw, gw′)p,q = (w, w′)p,q,w, w′ ∈Rp+q.Clearly the group O(p), the isometry group of (·, ·)p on Rp, can be regarded as asubgroup of O(p, q) by letting it act on the x component of w = (x, y). Similarly,O(q), the isometry group of (·, ·)q on Rq, can be considered a subgroup of O(p, q).Combining these, we get an embedding of O(p)×O(q) ⊆O(p, q), as the subgroup ofO(p, q), which stabilizes the decomposition Rp+q ≃Rp ⊕Rq implicit in the notationw = (x, y).

The group O(p) × O(q) is clearly a compact subgroup of O(p, q). It iswell known and not hard to see that O(p) × O(q) is a maximal compact subgroupof O(p, q).Let Mm,n(F) denote the space of m × n matrices over a field F; if m = n,we shall abbreviate Mm,m(F) by Mm(F).

We denote by o(p, q) the Lie algebra ofO(p, q), the space of (p+q)×(p+q) matrices T ∈Mp+q(R), which are generators ofone-parameter subgroups {esT | s ∈R} of O(p, q). By differentiating the relation(esT w, esT w′)p,q = (w, w′)p,q,we obtain the descriptiono(p, q) =T ∈Mp+q(R) (T w, w′)p,q + (w, T w′)p,q = 0for all w, w′ ∈Rp+q=ABBtC A = −At ∈Mp(R); C = −Ct ∈Mq(R); B ∈Mp,q(R).

(2.1)Here At denotes the transpose of the matrix A.

4R. E. HOWE AND E. C. TANConsider the light coneX0p,q = X0 =nw ∈Rp+q −{0}r2p,q(w) = 0o=n(x, y) ∈Rp+q −{0}r2p(x) = r2q(y)o.

(2.2)Observe that we exclude the origin from X0.The action of O(p, q) on Rp+q preserves the light cone X0. It is well known (inparticular, it is a direct consequence of Witt’s Theorem [Ja]) that X0 consists ofa single orbit for O(p, q), that is, the action of O(p, q) on X0 is transitive.

Thus ifw1 is a specified point in X0 and Q1 ⊆O(p, q) is the subgroup that stabilizes w1,then the map g →gw1 defines a bijection (in fact, a diffeomorphism)α : O(p, q)/Q1 ≃X0.Let P1 ⊆O(p, q) be the stabilizer of the line Rw1 through w1. Then P1 is aparabolic subgroup [Kn] of O(p, q), and α(P1) = R×w1; more precisely, the mapp →˜α(p) where pw1 = ˜α(p)w1 defines a group isomorphism˜α : P1/Q1 ≃R×.

(2.3)The second description of X0 in (2.2) allows us to analyze the action of O(p)×O(q)on X0. LetSp−1 = {x ∈Rp | (x, x)p = 1}be the unit sphere in Rp.

It is a homogeneous space for O(p). We have on X0 ananalog of “polar coordinates”.Scholium 2.1.

The mappingβ : X0 →Sp−1 × Sq−1 × R×+(2.4)defined byw = (x, y) → xr2p(x)1/2 ,yr2q(y)1/2 , r2p(x)1/2!is a diffeomorphism. The level sets of r2p(x) are O(p) × O(q) orbits, equivalent viaa scalar dilation to Sp−1 × Sq−1.The action of O(p, q) on Rp+q can be used to define an action of O(p, q) onfunctions on Rp+q by the familiar formulaρ(g)(f)(w) = f(g−1w),g ∈O(p, q), w ∈Rp+q,(2.5)for a function f on Rp+q.

Precisely the same formula defines an action on functionson any set X ⊆Rp+q that is invariant under O(p, q). Furthermore, the restrictionmapping rX : f →f1 taking a function f to its restriction to X, clearly commuteswith the O(p, q) action.

This means that if we have a function f on X, we cancompute ρ(g)(f) by extending f to some function ˜f on all of Rp+q, computingρ(g)( ˜f), then restricting back to X. This compatibility justifies at least partiallyour abuse of notation in failing to specify on what set f is defined in formula (2.5).We will be interested in taking X = X0, the light cone.

DEGENERATE PRINCIPAL SERIES5If we differentiate the action (2.5) along one-parameter subgroups of O(p, q), weobtain a representation of the Lie algebra o(p, q) on functions via differential oper-ators. Straightforward computation shows that the space of operators representingo(p, q) is spanned by the first-order operators:k1jk = xj∂∂xk−xk∂∂xj,1 ≤j < k ≤p,k2lm = yl∂∂ym−ym∂∂yl,1 ≤l < m ≤q,pjl = xj∂∂yl+ yl∂∂xj,1 ≤j ≤p, 1 ≤l ≤q.

(2.6)Here the k1jk describe the action of the Lie subalgebra o(p) ⊆o(p, q) correspondingto the subgroup O(p) of O(p, q). Similarly the k2lm describe the action of o(q).

Thepjl span a complement to o(p) ⊕o(q) in o(p, q), corresponding to the off-diagonalmatrix B in formula (2.1). We denote the span of the pjl by p.We propose to study spaces of homogeneous functions on the light cone X0.Thus for a ∈C denote by Sa(X0) the spaceSa(X0) =f ∈C∞(X0) f(tw) = taf(w), w ∈X0, t ∈R×+(2.7)of smooth functions on X0 homogeneous of degree a under dilations by positivescalars.

Since O(p, q) commutes with scalar dilations, it is clear that Sa(X0) willbe invariant under the action (2.5) of O(p, q). Indeed there is no need to consideronly positive dilations.

We could further define two subspacesSa±(X0) =nf ∈Sa(X0) f(−w) = ±f(w)o.It is clear thatSa(X0) = Sa+(X0) ⊕Sa−(X0)(2.8)and that each of Sa+(X0) and Sa−(X0) are individually invariant under O(p, q).Thus it would be possible, and might seem advisable, to study the Sa+(X0) andthe Sa−(X0) separately. However, we will find it convenient and enlightening tolump Sa±(X0) together in the space Sa(X0).Before proceeding to explicit computations, we observe that the spaces Sa±(X0)are naturally identifiable as O(p, q) modules to certain induced representations, of atype known informally as “degenerate principal series”.

Recall the homomorphism˜α :P1 →R× of formula (2.3).For a ∈C, and a choice ± of sign, define aquasicharacter (= homomorphism to C×) of P1 byψ±a (p) =(˜α(p)aif˜α(p) > 0,± |˜α(p)|aif˜α(p) < 0. (2.9)Define a mapδ : Sa±(X0) →C∞(G)(2.10)by the formulaδ(f)(g) = ρ(g)(f)(w1) = f(g−1w1).

6R. E. HOWE AND E. C. TANHere w1 ∈X0 is the point used to define P1.Straightforward formal checking shows thatδ(f)(pg) = ψ±a (p)−1δ(f)(g) ,p ∈P1, g ∈G,δ(f)(gh) = δ(ρ(h)f)(g) .

(2.11)These formulas, plus the definition of P1, imply that δ defines an equivalence ofO(p, q) representations between Sa±(X0) and the representation indGP1(ψ±a )−1 in-duced from the character (ψ±a )−1 of P1 [Kn]. (Here we are using unnormalizedinduction.) If q = 1 and we choose + from ±, these representations are the spheri-cal principal series of O(p, 1) [Kn].Now let us investigate the structure of the Sa(X0) as O(p, q) modules.

Firstconsider the action of the compact subgroup O(p) × O(q).The decomposition(2.4) of X0 shows that any function in Sa(X0) is determined by its restriction toβ−1(Sp−1 × Sq−1 × {1}). More exactly, restriction to β−1(Sp−1 × Sq−1 × {1}) ofelements of Sa(X0) defines an isomorphism of O(p) × O(q) modules.The classical theory of spherical harmonics [Ho3, Vi, Zh] describes C∞(Sp−1) asa representation of O(p).

It is the (topological) direct sum of the finite-dimensionalspaces Hm(Rp) consisting of the (restrictions to Sp−1 of the) harmonic polynomialsof degree m, m ∈Z+;C∞(Sp−1) ≃Xm≥0Hm(Rp). (2.12)The space Hm(Rp) consists of polynomials of degree m which are annihilated bythe Laplace operator∆p =pXj=1∂2∂x2j.Let Pm(Rp) denote the space of polynomials of degree m on Rp.

Then the Laplaceoperator maps Pm(Rp) to Pm−2(Rp); that is,∆p : Pm(Rp) →Pm−2(Rp),(2.13)with kernel Hm(Rp). It is part of the theory of spherical harmonics that the map-ping (2.13) is surjective; hencedim Hm(Rp) = dim Pm(Rp) −dim Pm−2(Rp)=m + p −1p −1−m + p −3p −1.If p > 1, this is always nonzero, but if p = 1, it is zero for m ≥2.

Thus althoughthe sum (2.12) always is nominally over all of Z+, for p = 1, it is effectively onlyover m = 0, 1. Each Hm(Rp) defines an irreducible representation of O(p), and therepresentations defined by two different Hm(Rp) are inequivalent.Combining the decompositions (2.12) for p and for q, we obtain a decompositionC∞(Sp−1 × Sq−1) ≃Xm,n≥0Hm(Rp) ⊗Hn(Rq)(2.14)

DEGENERATE PRINCIPAL SERIES7of the functions on Sp−1 × Sq−1 into irreducible inequivalent subspaces for O(p) ×O(q). The above remarks about restriction to β−1(Sp−1 × Sq−1 × {1}) show thatequation (2.14) also yields a description of the O(p) × O(q)-module structure ofSa(X0).The main point to observe is that we can adjust the homogeneity offunctions by multiplication by powers of r2p (or r2q) without changing the way theytransform under O(p) × O(q).

Thus we can define an embeddingja,m,n = ja : Hm(Rp) ⊗Hn(Rq) →Sa(X0)(2.15)by the formulaja(h1 ⊗h2)(x, y) = h1(x)h2(y)r2p(x)b,where h1 ∈Hm(Rp), h2 ∈Hn(Rq), and b is chosen so that the total degree ofhomogeneity is a, that is, m + n + 2b = a. In formula (2.15) we are implicitlyrestricting h1h2r2bpto X0.

The following statement summarizes our discussion ofthe O(p) × O(q) action.Lemma 2.2. As an O(p)×O(q) module, the space Sa(X0) decomposes into a directsumSa(X0) ≃Xm,n≥0ja(Hm(Rp) ⊗Hn(Rq))(2.16)of mutually inequivalent irreducible representations.We will refer to the spaces ja(Hm(Rp) ⊗Hn(Rq)) as the K-types of Sa(X0).We remind the reader that in case p (or q) equals 1, the sum over m (or n) is limitedto m (or n) equal to 0 or 1.Lemma 2.2 tells us the structure of Sa(X0) as an O(p)×O(q) module.

Since theK-types all define distinct O(p)×O(q) subrepresentations, any O(p)×O(q)-invariantsubspace of Sa(X0) will be a sum of the K-types it contains. This will in particularbe true of any O(p, q)-invariant subspace of Sa(X0).Thus, to understand theO(p, q)-module structure of Sa(X0), we need to see how O(p, q) transforms one K-type to another.

For this purpose, it is helpful to represent the K-types graphically,as points in the plane R2. Precisely, we will represent ja(Hm(Rp) ⊗Hn(Rq)) as thepoint with integer coordinate (m, n) in the positive quadrant (R+)2 in R2.

Thus theset of all K-types is matched to the set of all integer points in the positive quadrant,as illustrated by Diagram 2.17 (see next page). (This picture is for p, q ≥2, whichwe will assume for the present.

We will treat the degenerate case q = 1 later.) Wehave used two different symbols to mark the points to remind the reader of thedecomposition of Sa(X0) into Sa±(X0).

The dots indicate K-types in Sa+(X0)(the parity of m + n is even), and the crosses indicate K-types in Sa−(X0) (theparity of m+n is odd). Sometimes we will indicate a K-type ja(Hm(Rp)⊗Hn(Rq))simply by its associated coordinates (m, n).

8R. E. HOWE AND E. C. TAN✲✻sssssssssssssss×××××××××××××××xyLegend:sK -types of Sa+(X0)×K -types of Sa−(X0)Diagram 2.17To go beyond the K-type structure of Sa(X0), we will investigate the effect ofapplying elements from the subspace p of the Lie algebra o(p, q) to the K-types.Since the K-types are individually invariant under o(p) ⊕o(q), any collection ofK-types such that any one of them is mapped by p into the span of the others willdefine (i.e., its span will be) an o(p, q) submodule of Sa(X0).

An early theorem ofHarish-Chandra [HC2] guarantees that the closure of this span in Sa(X0) will be anO(p, q) submodule. Thus our computation of the action of p on individual K-typesis in principle (and will turn out to be in practice) sufficient for understanding thesubmodule structure of Sa(X0).The operators of p are described in the formula (2.6).

A computation, whichwill be carried out in §6.1, establishes the following rules.Lemma 2.3. For each pair (m, n), there are mapsT ±,±m,n : p ⊗(Hm(Rp) ⊗Hn(Rq)) →Hm±1(Rp) ⊗Hn±1(Rq),which are independent of a and which are nonzero as long as the target space isnonzero, such that the action of z ∈p on the K-type ja(Hm(Rp) ⊗Hn(Rq)) isdescribed by the formulaρ(z)ja(φ) =(a −m −n)ja(T ++mn (z ⊗φ))+ (a −m + n + q −2)ja(T +−mn (z ⊗φ))+ (a + m −n + p −2)ja(T −+mn (z ⊗φ))+ (a + m + n + p + q −4)ja(T −−mn (z ⊗φ)),(2.18)where φ ∈Hm(Rp) ⊗Hn(Rq).Proof.

See §6.1.In terms of the graphical representation (see Diagram 2.17) of the K-types, wemay picture this formula as follows. Starting from a given point (m, n), the actionof p potentially can take us to any of the diagonally adjacent points (m± 1, n± 1).Whether we can, in a given Sa(X0), actually proceed from (m, n) to a given one ofthe points (m ± 1, n ± 1) depends on whether the coefficient of the operator T ±±mn

DEGENERATE PRINCIPAL SERIES9(e.g., a −m −n for T ++mn ) in the formula (2.18) is nonzero or not. We can representthe situation graphically as in Diagram 2.19.✠❅❅❅❅❘✒❅❅❅❅■(m, n)(m −1, n −1)(m + 1, n −1)(m + 1, n + 1)(m −1, n + 1)T −+mnT ++mnT +−mnT −−mnDiagram 2.19The four corner points in Diagram 2.19 are the K-types we may hope to reachfrom (m, n) by the action of p; whether we actually can get to a given one of thepoints depends on whether the transition coefficient A±±p,q,a(m, n) given byA++p,q,a(m, n) = a −m −n,A+−p,q,a(m, n) = a −m + n + q −2,A−+p,q,a(m, n) = a + m −n + p −2,A−−p,q,a(m, n) = a + m + n + p + q −4,(2.20)is nonzero.

We shall omit the subscripts in A±±p,q,a if it is clear from context.Remark. We see that from a given (m, n) by arbitrarily repeated action of p wecan never hope to get to all other points (m′, n′)—only to those such that m′ + n′has the same parity as m+ n. This reflects the fact, already observed, that Sa(X0)breaks up into the two invariant subspaces Sa±(X0).We see that if a is not an integer, then the transition coefficients A±± are neverzero for any (m, n) in (Z+)2.

From this and Diagram 2.19, we see that, startingfrom any K-type ja(Hm(Rp) ⊗Hn(Rq)), we may by sufficiently many applicationsof elements from p eventually reach any other K-type corresponding to (m′, n′),providing m′ + n′ has the same parity as (m, n).Combining this with the de-scriptions (2.17), we can make our first conclusion concerning reducibility of theSa(X0).Proposition 2.4. If a is not an integer, then each of Sa±(X0) is an irreducibleO(p, q) module.Consider now what happens when a is an integer.

Suppose a ≥0, and considera K-type (m, n) such that m + n = a.This means A++ = 0, so we cannotmove from (m, n) to (m + 1, n + 1) by acting by p.We can perhaps move to(m′, n′) = (m + 1, n −1) or to (m′, n′) = (m −1, n + 1), but from these points weagain cannot move to the upper right, as seen in terms of Diagram 2.19 since again

10R. E. HOWE AND E. C. TANm′ + n′ = a.

We can see that the action of p, no matter how many times repeated,will never take us above the line m + n = a. Thus the set of K-types (m, n) withm + n ≤a and m + n ≡a(mod 2) form an o(p, q) submodule of Sa(X0).We interpret this using our graphical representation of K-types as follows.

Wethink of the line a = x + y in R2 as being a barrier, which moves as a varies. Whenthis moving line passes through integer points in the positive quadrant, it blocksattempts to move past it from those K-types (of the correct parity); thus, trappingall K-types below it in an o(p, q) submodule.

This is illustrated by Diagram 2.21.✲✻sssssssssssssssssssssssss❅❅❅❅❅❅❅❅✠✠✠xaayDiagram 2.21Similar reasoning applies to each of the transition coefficients A±±. Each ofthem can be thought of as defining a line in R2, namely, the set of (x, y) ∈R2 thatsatisfy A±±(x, y) = 0.

We will abuse notation and use the symbol A±± to referto the line or barrier defined by the zero locus of the linear functional A±±(x, y)defined in (2.20). These lines move as a varies, and when they pass through integerpoints (which they all do simultaneously, when a is an integer), they trap the K-types on one side and so give rise to a submodule.

The position of these barriersfor a typical a > 0 is given in the Diagram 2.22.

DEGENERATE PRINCIPAL SERIES11✲✻yx❅❅❅❅❅❅❅❅❅❅❅sss✠✠✠sss✒✒✒A++ : x + y = assaaA−−: x + y = −a −p −q + 4sss❅❅■❅❅■❅❅■A+−: y = x −a −q + 2sa + q −2sss❅❅❘❅❅❘❅❅❘A−+ : y = x + a + p −2sa + p −2Diagram 2.22The arrows on each barrier point in the direction of the submodule defined bythe barrier. Observe that the barriers A++ and A−−are parallel and the associatedarrows point in opposite directions.

Likewise the barriers A+−and A−+ are parallelto each other and perpendicular to the other two. As a varies, the barriers A++and A+−move so that they intersect the x-axis at points q −2 units apart, andthe barriers A++ and A−+ always intersect the y-axis at points p −2 units apart.Thus A++ never intersects A+−or A−+ inside the positive quadrant unless p or qequals 2.

Also, the barriers A++ and A−−never intersect the positive quadrant atthe same time. They coincide when a = 2 −p+q2 .

The barriers A+−and A−+ alsocoincide at this value of a. For 0 > a > −p −q + 4, neither of the barriers A++ orA−−intersects the positive quadrant.All submodules of Sa(X0) are accounted for by the barriers A±± and by thedecomposition (2.8).

The interactions between these effects yield several patternsof reducibility according to the parities of p and q. The details are quite easy to workout.

We will summarize the results, with diagrams illustrating typical situations.The main point to consider is which of the barriers A±± affect Sa+(X0) andwhich affect Sa−(X0). For example, suppose (m, n) ∈(Z+)2 satisfies A++(m, n) =0 while another point (m′, n′) ∈(Z+)2 satisfies A+−(m′, n′) = 0.

Then(m + n) −(m′ + n′) =a −(2m′ −a −q + 2)=2(a −m′ −1) + q.Hence the parity of (m + n) −(m′ + n′) is the same as the parity of q. Thus if q iseven, the barriers A++ and A+−both affect the same module of the pair Sa±, butif q is odd, one of the barriers affects Sa+, and the other affects Sa−.

Similarly, thepair of barriers A++ and A−+ affects the same module if p is even and different

12R. E. HOWE AND E. C. TANones if p is odd.

Also A+−and A−+ affect the same or different modules accordingas p + q is even or odd.We describe the various possibilities with a collection of diagrams with commen-taries. In all the diagrams below, the arrows on the barriers point toward the regionof K-types that define a submodule.Case OO: p odd, q odd.

In this situation, the barriers A+−and A−+ affect thesame one of the Sa±(X0) and A++ or A−−affects the other. For the record, wenote that A+−and A−+ affect Saε(X0) when ε = (−1)a+1, while A++ and A−−affect Saε(X0) when ε = (−1)a.

We describe the submodule structure for variousranges of values of a. (i) a ≥0.Here the module affected by the barriers A+−and A−+ decomposesinto three components: one irreducible submodule and two irreducible quotients,all infinite dimensional, illustrated by Diagram 2.23.✲✻ss❅■❅■❅■❅❘❅❘❅❘A−+A+−a + p −2a + q −2Diagram 2.23The other of Sa±(X0) decomposes into two components: one finite-dimensionalsubmodule and one infinite-dimensional quotient, illustrated by Diagram 2.24.✲✻✠✠✠❅❅❅❅❅❅❅ssaaA++Diagram 2.24(ii) 0 > a > −[(p+q)/2]+2.In this range, the structure of the module affectedby the barriers A+−and A−+ remains qualitatively the same as when a ≥0, butthe other module is irreducible.

DEGENERATE PRINCIPAL SERIES13(iii) a = −[(p + q)/2] + 2.For this value of a, the two barriers A+−and A−+coalesce, and there is a submodule consisting of K-types supported on the liney = x + (p −q)/2 and two quotient modules consisting of the K-types on one sideor the other of this line (see Diagram 2.25).✲✻❅❅❘❅❅❘❅❅❘❅❅■❅❅■❅❅■A+−= A−+ = 0sp −q2Diagram 2.25(iv) a = −(p + q)/2 + 1.For this value of a, the module affected by A+−and A−+ (which is Sa+(X0) if (p −q)/2 is odd and Sa−(X0) if (p −q)/2 is even)decomposes into the direct sum of two irreducible modules. This is because thereare no K-types in the region between the two barriers (see Diagram 2.26).✲✻❅❅❘❅❅❘❅❅■❅❅■ssA−+A+−(p −q)/2 −1(p −q)/2 + 1Diagram 2.26This is the only value of a that gives a decomposable module.

The other moduleremains irreducible. (v) a = −[(p + q)/2].This situation is dual to case (iii).

The constituent witha single line of K-types is now a quotient, with barriers trapping the K-types oneither side of this line in a submodule. See Diagram 2.27.

14R. E. HOWE AND E. C. TAN✲✻❅❘❅❘❅■❅■ssssssssssssA−+A+−(p −q)/2 −2(p −q)/2 + 2(p −q)/2Diagram 2.27Notice that if the module depicted by Diagram 2.26 in case (iv) is Sa+(X0), theone affected here is Sa−(X0), and vice versa.

The other module remains irreducible. (vi) −[(p + q)/2] > a ≥−(p + q) + 4.Here the picture is dual to case (ii).

Themodule affected by A+−and A−+ continues to have three constituents, but nowit has two submodules and one quotient (see Diagram 2.28). The other moduleremains irreducible.✲✻ss❅❘❅❘❅❘❅■❅■❅■A+−A−+−(a + q −2)−(a + p −2)Diagram 2.28(vii) a < −(p+q)+4.The module affected by A+−and A−+ has qualitativelythe same structure as in case (vi).

The other module now has two constituents: aninfinite-dimensional subrepresentation and a finite-dimensional quotient (see Dia-gram 2.29).

DEGENERATE PRINCIPAL SERIES15✲✻✒✒✒❅❅❅❅❅❅❅ss−(a + p + q −4)−(a + p + q −4)A−−Diagram 2.29Case OE: p odd, q even. In case OE, the barriers A+−and A++ affect one ofSa±(X0), and the barriers A−+ and A−−affect the other one.

For the record,A+−and A++ affect Saε(X0) if ε = (−1)a, while A−+ and A−−affect Saε(X0) ifε = (−1)a+1. Here is the description of submodule structure in various ranges ofthe value of a.

(i) a > 0.Here the module affected by A++ and A+−has three constituents:one finite-dimensional subrepresentation, one infinite-dimensional quotient, and onesubquotient in between, as illustrated by Diagram 2.30.✲✻✠✠❅■❅■A+−❅❅❅❅❅❅sssaa + q −2aA++Diagram 2.30The module affected by A−+ has two constituents (see Diagram 2.31).

16R. E. HOWE AND E. C. TAN✲✻s❅❘❅❘❅❘A−+a + p −2Diagram 2.31(ii) 0 > a > −(p+q)+4.In this range of a-values, the module affected by A−+retains the same qualitative structure as it has in case (i), but the other module isno longer affected by A++, so it also consists of only two constituents (see Diagram2.32).✲✻s❅■❅■❅■A+−a + q −2Diagram 2.32(iii) a < −(p + q) + 4.The situation is dual to that of case (i).

The moduleaffected by A+−continues qualitatively as in case (ii), while the module affectedby A−+ is now also affected by A−−and has three constituents, including a finite-dimensional quotient (see Diagram 2.33).

DEGENERATE PRINCIPAL SERIES17✲✻✒✒❅❘❅❘❅❅❅❅❅❅sss−(a + p −2)−(a + p + q −4)A−−A−+Diagram 2.33Case EO: p even, q odd. This case, mutatis mutandis, is essentially the same asCase OE.Case EE: p even, q even.

When p and q are both even, all the possible bar-riers affect the same one of Sa±(X0) (i.e., Saε(X0) if ε = (−1)a), and the otheris irreducible.The discussion and diagrams below will treat only the reduciblesummand. (i) a > 0.Here there are four constituents: one finite-dimensional subrepresen-tation, two quotients, and one subquotient in between, as illustrated by Diagram2.34.✲✻✠✠❅■❅■❅❘❅❘aa + q −2aa + p −2A++A−+A+−❅❅❅❅❅❅ssssDiagram 2.34(ii) 0 > a > −(p + q) + 4.Here, in the range where the barriers A++ and A−−have no effect, the description of submodules is the same as in cases OO(ii)–(vi).The most interesting things happen in the range −[(p+q)/2]+2 ≥a ≥−[(p + q)/2].

(iii) a < −(p + q) + 4.The picture is dual to case (i). There is one finite-dimensional quotient, two infinite-dimensional subrepresentations, and one con-stituent in between (see Diagram 2.35).

18R. E. HOWE AND E. C. TAN✲✻✒✒❅❅❘❅❅❘❅❅■❅❅■❅❅❅❅❅❅ssss−(a + p −2)−(a + q −2)−(a + p + q −4)A−−A−+A+−Diagram 2.35Remarks.

(a) Although there are considerable variations in the submodule struc-ture of Sa(X0) according to the parities of p and q, there is always a total of fiveconstituents for a > 0, one of which is finite dimensional. (b) The extra complexity when p + q is even of the submodule structure in therange −[(p + q)/2] + 2 ≥a ≥−[(p + q)/2 will acquire additional significance in §3where unitarity is investigated.

(c) Although we have only described the constituents of Sa(X0) as “finite dimen-sional” or “infinite dimensional”, the Diagrams 2.23, 2.28, 2.34, and 2.35 certainlysuggest that the constituent “in the middle” is smaller than the constituents “in thewings”. This difference in size could be made precise in terms of Gelfand-Kirillovdimension [Vo1] or N-spectrum [Ho2].Degenerate cases p ≥q = 1.

We will now indicate how the case of q = 1 differs fromthe picture developed above and also make some supplementary remarks about thecase q = 2, which is also somewhat exceptional.When q = 1, the light cone X0 is not connected. We can divide it into tworelatively open and closed pieces X0±, called nappes, defined byX0± = {(x, y) ∈X0 | y = ±|x|}.

(Note that y is just a real number when q = 1. )The two nappes X0± of thelight cone are each stabilized by a subgroup O+(p, 1) of index 2 in O(p, 1).

Wesee that any element of Sa±(X0) is determined by its restriction to X0+. Fur-thermore the function sign on X0, which takes the values ±1 on X0±, is clearlyinvariant under O+(p, 1) and is an eigenfunction of O(p, 1).

Evidently multiplyingby sign gives an O+(p, 1)-module isomorphism between the Sa±(X0). Thus whenq = 1, analyzing Sa(X0) as an O(p, 1) module is essentially the same as analyz-ing Sa+(X0) ≃Sa(X0+) as an O+(p, 1) module.

We will do this, following anabbreviated version of our agenda for the case of O(p, q), p, q > 1.

DEGENERATE PRINCIPAL SERIES19The mapβ : X0+ →Sp−1 × R×+,β(x, y) = (x/y, y),(2.36)where x ∈Rp −{ 0 } and y = r2p(x)1/2 ∈R×+, is a diffeomorphism. The maximalcompact subgroup of O+(p, 1) is O(p), which acts transitively on the set {(x, 1) ∈X0+} ≃Sp−1.

We can define embeddingsja,m = ja : Hm(Rp) →Sa(X0+),ja(h)(x, y) = h(x)ya−m. (2.37)Note that since y > 0 on X0+, complex powers of y make sense.

From decomposi-tion (2.36) and the theory of spherical harmonics, we know that the decompositionof Sa(X0+) as an O(p) module isSa(X0+) ≃Xm≥0ja(Hm(Rp)).To complete the picture we want to compute the action of p (see formulas (2.6))on the K-types ja(Hm(Rp)). The analogue of Lemma 2.3 isxj∂∂y + y ∂∂xj(hya−m)= (a −m)h+j ya−m−1 + (a + m + p −2)h−j ya−m+1= (a −m)T +j (h)ya−m−1 + (a + m + p −2)T −j (h)ya−m+1,whereh+j = xjh −(2m + p −2)−1 ∂h∂xjr2p,h−j = (2m + p −2)−1 ∂h∂xj,h ∈Hm(Rp).We note that h−j is harmonic, while h+j is the projection of xjh into Hm+1(Rp).The principles behind this calculation are discussed in §6.1.

It follows easily thatSa(X0+) is always irreducible except when a is an integer, with either a ≥0 ora ≤−p + 1, in which case Sa(X0+) has two constituents, one of which is finitedimensional. In what we hope will be a self-evident analogy with the situation forq > 1, we illustrate this by Diagram 2.38.ssssa ≤−p + 1a ≥0iha−(a + p −2)Diagram 2.38For −p + 1 < a < 0, the representations Sa(X0+) are irreducible.

20R. E. HOWE AND E. C. TANAlthough the general picture presented in Diagrams 2.23–2.35 is valid whenq = 2, this case is also somewhat exceptional.

Although the Hn(R2) are nonzerofor all n ≥1, their dimensions, instead of growing with n, remain fixed at 2 (forn = 0, Hn(Rq) is the trivial representation for all q and has dimension 1). Moresignificantly, whereas the representations Hn(Rq), n ≥1, q ≥3, of O(q) remainirreducible when restricted to the special orthogonal group SO(q), the representa-tions Hn(R2), n ≥1, decompose into two one-dimensional representations whenrestricted to SO(2), which is isomorphic to T, the unit circle.

This suggests thatone consider varying the picture presented above by restricting the representationsSa(X0) to the identity component O0(p, q) of O(p, q). For p, q ≥3 this has noeffect.

When q = 2 (and p > 2), however, the pictures presented above for O(p, 2)must be “folded out” along the x-axis to yield pictures appropriate for O0(p, q).Precisely the Diagrams 2.30, 2.31, and 2.33 now should look as in Diagrams 2.39,2.40, and 2.41 respectively.✠✠✠❅■❅■❅■❅■❅❅❅❅❅❅❅❅❅−aaA++A+−aDiagram 2.39❅❘❅❘✒✒❅❅❅❅❅−(a + p −2)a + p −2A−+Diagram 2.40

DEGENERATE PRINCIPAL SERIES21✒✒✒✒❅❘❅❘❅❘❅❅❅❅❅❅❅❅❅a + p −2−(a + p −2)A−−A−+Diagram 2.41We observe that in Diagram 2.22, the gap between the points where A++ andA+−intersect the x-axis is q −2, so when q = 2, these points coincide, and so whenreflected across the x-axis, these barriers make crossing straight lines.Similarremarks apply to A−−and A−+.If p is even, then the relevant diagrams are 2.34 and 2.35. The modified versionof Diagram 2.34 is Diagram 2.42.✠✠✠✠❅■❅■❅■❅■✒✒❅❘❅❘❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅−aaA++A+−a−(a + p −2)a + p −2A−+Diagram 2.42Thus in this case, we have six constituents in all, with Hasse diagram [Sta] as

22R. E. HOWE AND E. C. TANillustrated by Diagram 2.43.❅❅❅❅❅❅❅❅ttttttDiagram 2.43We omit the diagram corresponding to Diagram 2.35, which would be dual toDiagram 2.42.

The main effect of the restriction is that representations that haveno K-types along the x-axis, that is, that contain no fixed vectors for O(2), breakup into two summands symmetrically placed across the x-axis. However, represen-tations whose K-type region includes part of the x-axis remain irreducible.

Rein-forcing this splitting is the fact that when q = 2, the barriers A++ and A+−(andthe barriers A−−and A−+) intersect on the x-axis, leaving no gap for K-typesof the intermediate constituent in Diagram 2.30. Hence this constituent becomestwo on restriction to O0(p, 2).

The representations of O0(p, 2) whose K-types arerestricted to only one side of the x-axis are of the type known as holomorphic orhighest weight modules [EHW, Br].If p = 2 and q = 2, then Diagram 2.42 should be further folded out along they-axis. Since O(2, 2) is just isogenous to SL(2, R) × SL(2, R), we will not dwell onthis example.3.

UnitarityIn this section we will investigate which of the subquotients of the Sa(X0) de-scribed in §2 can define unitary representations, that is, possess an O(p, q)-invariantpositive-definite Hermitian form. As with the calculations of submodule structure,the results are primarily based on the formulas of Lemma 2.3 for the action ofp ⊆o(p, q).Since the K-types ja(Hm(Rp)⊗Hn(Rq)) are irreducible modules for the compactgroup O(p) × O(q), each one admits an O(p) × O(q)-invariant Hermitian innerproduct, and this is unique up to multiples.

For this discussion we will fix an innerproduct ⟨·, ·⟩m,n on each space Hm(Rp) ⊗Hn(Rq) by the geometric recipe⟨φ1, φ2⟩m,n =ZSp−1×Sq−1 φ1(z)φ2(z) dz,(3.1)where φj ∈Hm(Rp)⊗Hn(Rq) and dz indicates the O(p)×O(q)-invariant probabilitymeasure on Sp−1 × Sq−1. We will push forward this inner product by ja to obtainan inner product on the K-type ja(Hm(Rp) ⊗Hn(Rq)).

We call the collection ofinner products so obtained the standard inner products on the K-types.If ⟨·, ·⟩is an O(p, q)-invariant Hermitian form on Sa(X0) (or on some constituentof Sa(X0)), the restriction of ⟨·, ·⟩to any K-type must be a multiple of the standard

DEGENERATE PRINCIPAL SERIES23inner product on that K-type:⟨ja(f1), ja(f2)⟩= cm,n⟨f1, f2⟩m,n,(3.2)where fj ∈Hm(Rp) ⊗Hn(Rq) and cm,n is an appropriate real number, which mustbe positive if ⟨·, ·⟩is to be positive definite. Furthermore the K-types must bemutually orthogonal, so that ⟨·, ·⟩is determined by the numbers cm,n.According to an early theorem of Harish-Chandra [HC2], a positive-definite innerproduct ⟨·, ·⟩on Sa(X0) will be O(p, q) invariant if and only if the operators fromo(p, q) are formally skew adjoint with respect to ⟨·, ·⟩.

Equation (3.2) guaranteesthis will be so for operators from o(p)⊕o(q). Thus to determine invariance of ⟨·, ·⟩wejust need to check skew-adjointness of the operators from the subspace p ⊆o(p, q).From formulas (2.18) and (3.2), we see that this amounts to the equations(a)0 =(a −m −n)cm+1,n+1⟨T ++m,n(z ⊗φ), φ′⟩m+1,n+1+ (a + m + n + p + q −2)cm,n⟨φ, T −−m+1,n+1(z ⊗φ′)⟩m,n,(b)0 =(a −m + n + q −2)cm+1,n−1⟨T +−m,n(z ⊗φ), φ′′⟩m+1,n−1+ (a + m −n + p)cm,n⟨φ, T −+m+1,n−1(z ⊗φ′′)⟩m,n,(3.3)where z ∈p, φ ∈Hm(Rp) ⊗Hn(Rq), φ′ ∈Hm+1(Rp) ⊗Hn+1(Rq), and φ′′ ∈Hm+1(Rp) ⊗Hn−1(Rq).

Here a means the complex conjugate of a.Equations (3.3) can be substantially simplified by an elementary observation,which will be justified in §6.Lemma 3.1. When Re a = −[(p + q)/2] + 1, that is, for a = −[(p + q)/2] + 1 + is,s ∈R, the Hermitian inner product which equals the standard inner product oneach K-type, is O(p, q) invariant.Remark.

More generally, one can show that if φ1 in formula (3.1) is taken to be-long to Sα(X0) and φ2 is taken from Sα′(X0), where α + α′ = 2 −(p + q), then(3.1) defines an O(p, q)-invariant pairing between Sα(X0) and Sα′(X0), which arethus canonically mutual Hermitian duals. This duality explains the symmetry instructure among the Sα(X0), as discussed in §2.Proof.

See §6.1.If we take a = −[(p + q)/2] + 1 in formula (3.3), we see that Lemma 3.1 implies(a) ⟨T ++mn (z ⊗φ), φ′⟩m+1,n+1 = ⟨φ, T −−m+1,n+1(z ⊗φ′)⟩m,n,(b) ⟨T +−mn (z ⊗φ), φ′′⟩m+1,n−1 = ⟨φ, T −+m+1,n−1(z ⊗φ′′)⟩m,nfor all possible choices of z, φ, φ′, φ′′ as in (3.3). Given these relations, a reinspec-tion of formulas (3.3) shows they are equivalent to the purely numerical equations((a −m −n)cm+1,n+1 + (a + m + n + p + q −2)cm,n = 0,(a −m + n + q −2)cm+1,n−1 + (a + m −n + p)cm,n = 0or(A++(m, n)cm+1,n+1 + A−−(m + 1, n + 1)cm,n = 0,A+−(m, n)cm+1,n−1 + A−+(m + 1, n −1)cm,n = 0.

(3.4)

24R. E. HOWE AND E. C. TANLet us explore the implications of equations (3.4).

Consider first the case whenthe Sa±(X0) are irreducible. Then the quotientm + n −aa + m + n + p + q −2must at least be real for all possible choices of m and n. Setting˜a = a + p + q2−1,we can rewrite this ratio asm + n + δ −˜am + n + δ + ˜a = (m + n + δ −Re ˜a) −i Im ˜a(m + n + δ + Re ˜a) −i Im ˜a,(3.5)whereδ = p + q2−1.It is easy to convince oneself that the ratio (3.5) can be real for large m, n only ifeither Re ˜a = 0 or Im ˜a = 0.

The case of Re ˜a = 0 is covered by Lemma 3.1. Wewill call the set of a with Re ˜a = 0 the unitary axis.

If Im ˜a = Im a ̸= 0, we know(see Proposition 2.4) that Sa±(X0) is irreducible; hence for it to be Hermitian,expression (3.5) would have to be real for all m, n. This leads us to the followingconclusion.Lemma 3.2. In order for Sa(X0) to have any Hermitian constituents, either amust be on the unitary axis, or a must be real.

Conversely, for a on the unitaryaxis, Sa(X0) is unitary, and for a real, all components of Sa(X0) are Hermitian(i.e., they possess an invariant Hermitian form).The final assertion follows from the observation that if a is real then we canstart at a given K-type (m0, n0) in any constituent of Sa(X0), set cm0,n0 = 1, anduse equations (3.4) to determine cm,n for other K-types in that constituent. Itshould be checked that this procedure is consistent, i.e., that it results in a well-defined cm,n on each K-type.For this, it suffices to check that the transitionsaround squares, like (m, n) −→(m + 1, n + 1) −→(m + 2, n) and (m, n) −→(m + 1, n −1) −→(m + 2, n), predict the same value for the ratio of the start andfinish (here cm+2,n/cm,n).

This is a short computation. Here, as in the analysis ofirreducibility, it is germane to note that the transition coefficients A++ and A−−are unchanged by the transitions (m, n) −→(m, n) ± (1, −1); likewise A+−andA−+ are unchanged under (m, n) −→(m, n) ± (1, 1).Our goal is now to investigate for which constituents of Sa(X0) the processdescribed in the previous paragraph will yield positive values for all cm,n in theconstituent.

Looking again at equations (3.4), we can easily see and formulate thefollowing rule.Criterion 1. Let a be real.

A given constituent of Sa(X0) is unitary if and onlyif the ratios(i)A++(m, n)A−−(m + 1, n + 1)and(ii)A+−(m, n)A−+(m + 1, n −1)(3.6)

DEGENERATE PRINCIPAL SERIES25are negative for all pairs of K-types (m, n) and (m + 1, n + 1) or (m, n) and (m +1, n −1) contained in the constituent.We proceed to investigate the nature of this criterion in order to obtain anexplicit list of the unitary constituents.Consider again our graphical representation of the K-types as integer points inthe positive quadrant, together with barrier lines corresponding to the zeroes of thelinear functions A±±. We illustrate the situation for a typical a > 0 in Diagram3.7.✲✻❅❅❅❅❅❅❅❅❅❅A−−+−A++−++−A−++−A+−rrrraa + q −2aa + p −2Diagram 3.7In this diagram we have indicated, for each transition coefficient, the side ofits barrier where it is positive and the side where it is negative.

We see that, inthe positions illustrated, both A+−and A−+ are positive in the region betweentheir barriers and likewise for A++ and A−−. When a < 0 and the A+−barrieris above the A−+ barrier, both A+−and A−+ are negative in the region betweenbarriers.

In general, we may say that A+−and A−+ have the same sign in theregion between barriers, and have opposite signs in the regions outside the barriers,similarly for A++ and A−−. Thus, if (m, n) and (m+1, n−1) both lie between theA+−and A−+ barriers, or more precisely if (m, n) lies above the A+−barrier and

26R. E. HOWE AND E. C. TAN(m + 1, n −1) lies below the A−+ barrier (or vice versa), then the ratio (3.6)(ii) ispositive and unitarity cannot hold.

On the other hand, if (m, n) lies above the A+−barrier and (m + 1, n −1) also lies above the A−+ barrier (or both lie below), thenthe transition from (m, n) to (m + 1, n −1) is consistent with unitarity. Similarremarks apply to the pair A++, A−−.We can apply these considerations to determine the unitarity of constituents ofthe Sa(X0).

As for the discussion of submodule structure in §2, it is convenient todistinguish cases according to the parities of p and q. Before beginning we note that,using the notation ˜a as in formula (3.5), the same argument that establishes Lemma3.1 shows that the modules S˜a(X0) and S−˜a(X0) are mutual Hermitian duals.

Thiswill be shown in §6.1. It follows that in considering questions of unitarity, it willsuffice to take ˜a ≥0.Case OO: p odd, q odd.

From the unitary axis (˜a = 0) until ˜a = [(p + q)/2] −1(i.e., a = 0), neither of the barriers A++ or A−−intersects the positive quadrant.Hence unitarity depends only on the barriers A+−and A−+. Their position when˜a = 0 is illustrated in Diagram 2.26.Let ε0 denote the choice of ± such thatSaε0(X0) is reducible at ˜a = 0, and let ε1 denote the other choice.

For the record,we note that ε0 is + if (p −q)/2 is odd and is −if (p −q)/2 is even.At ˜a = 0 none of the K-types of Saε0(X0) lie between the barriers A+−and A−+,which are located on the two lines L±1 defined by equations y −x = [(p−q)/2]± 1;however, there are K-types lying on these two lines. As ˜a increases from 0, thebarrier A+−moves to the right and A−+ moves left.Hence, for 0 < ˜a < 2,transitions between L1 and L−1 cross both barriers and so violate unitarity.

For˜a ≥2, transitions between L1 and L−1 cross neither barrier and so again violateunitarity. Hence the full module Saε0(X0) is never unitary when ˜a > 0, and atpoints of reducibility the constituent with K-types between the barriers cannot beunitary.

The other two constituents, however, are easily seen to be unitary, sinceall their transitions take place on the same side of both barriers.The module Saε1(X0) is irreducible at ˜a = 0 and remains so for 0 < ˜a < 1.For this range of a, the only K-types between the barriers are those on the line L0defined by y −x = (p −q)/2. One checks that transitions in either direction fromL0 are consistent with unitarity.

At ˜a = 1 the barriers coincide on L0 (see Diagram2.25), and the K-types on this line span a constituent of Saε1(X0). There are notransitions transverse to the barriers from L0, so this constituent is unitary; likewisethe other constituents are unitary, so all three are unitary.

For ˜a > 1 the line L0 ison the opposite side of the barriers A+−and A−+, from where it was when ˜a < 1,so the transitions from L0 violate unitarity. Hence unitarity for Saε1(X0) stops at˜a = 1.

Also, for points of reducibility beyond ˜a = 1, the same reasoning applies asto Saε1(X0).In the range a ≥0, when there may be a finite-dimensional component, oneeasily checks that the trivial representation at a = 0 is unitary, but no other finite-dimensional representations are so.In sum, the unitarity properties of the Sa±(X0) in this case are summed up inDiagrams 3.8 and 3.9. For Diagram 3.9, we note that (a) and (b) depict Saε(X0)when ε = (−1)a+1, while (c) and (d) depict Saε(X0) when ε = (−1)a.

DEGENERATE PRINCIPAL SERIES27Saε0(X0) :unitary axisRe ˜a = 0s✒reducible pointat ˜a = 0Saε1(X0) :unitary axisRe ˜a = 0✻Diagram 3.8. Parameter values for unitarity of the full modules Sa±(X0).✲✻❅❘❅❘❅❘❅■❅■❅■A+−= A−+sp −q2unitaryunitary✲✻ss❅■❅■❅■❅❘❅❘❅❘A−+A+−non-unitaryunitaryunitary(a) at ˜a = 1(b) ˜a ∈Z, ˜a > 1✲✻s❅❅❅❅❅❅nonunitaryA++unitary:trivialrepresentation✲✻✠✠✠❅❅❅❅❅❅ssaaA++nonunitarynon-unitary(c) a = 0(d a ∈Z, a > 0)Diagram 3.9.

Unitarity at points of reducibility.Case OE: p odd, q even. In this case, when ˜a = 0, the barriers do not passthrough integral points, so the representations S−δ±(X0) are irreducible (recallthat δ = [(p + q)/2] −1).

Each has K-types between the barriers: one on the

28R. E. HOWE AND E. C. TANline L1/2 defined by y −x = [(p −q)/2] + 12 and the other on the line L−1/2,defined by y −x = [(p −q)/2] −12 (see Diagram 3.10).

As ˜a increases from 0,the barriers approach these lines and coincide with them for ˜a = 12. One checksthat for 0 < ˜a < 12 transitions from these lines is consistent with unitarity, butas ˜a passes 12, the barriers move past the lines L±1/2, and then transition fromthese lines does violate unitarity.

Hence for ˜a > 12, the full modules S˜a±(X0) canno longer be unitary. At points of reducibility for ˜a > 12, one can check that theconstituent with K-types to the right of A+−and the one with K-types to the leftof A−+ is unitary.

Again the only unitary finite-dimensional unitary constituent isthe trivial representation. Diagrams 3.11 and 3.12 illustrate the main conclusions.For Diagram 3.12, we note that A++ and A+−affect Saε(X0) if ε = (−1)a, whileA−+ and A−−affect Saε(X0) if ε = (−1)a+1.✲✻ssssssssA−+A+−p−q2−1p−q2+ 1Diagram 3.10.

˜a = 0.unitary axisRe ˜a = 0✻˜a ∈(−12, 12)Diagram 3.11. Parameter values of unitarity of the full modules Sa±(X0).

DEGENERATE PRINCIPAL SERIES29✲✻s❅■❅■❅■A+−(p−q)2+ 12unitaryunitary✲✻s❅❘❅❘❅❘A−+(p−q)2−12unitaryunitary(a) ˜a = 12✲✻s❅❘❅❘❅❘A−+˜a + (p−q−2)2nonunitaryunitary✲✻s˜a −(p−q+2)2❅■❅■❅■A+−unitarynonunitary(b) ˜a ∈12 + Z, ˜a > 12(c) ˜a ∈12 + Z, ˜a > 12, a < 0✲✻❅■❅■❅■ss❅❅❅❅❅❅unitarynonunitaryA++A+−q −2unitary:trivialrepresentation✲✻✠✠✠❅■❅■❅■nonunitaryunitarynon-unitaryA++A+−aa❅❅❅❅❅❅(d) a = 0(e) ˜a ∈12 + Z, a > 0Diagram 3.12. Unitarity of constituents at points of reducibility.Case EO: p even, q odd.

This case is essentially the same as case OE. We willnot give the details.Case EE: p even, q even.

Here the discussion is essentially the same as for CaseOO, except that, when a ≥0, at a point of reducibility, three barriers affect the

30R. E. HOWE AND E. C. TANsame Sa±(X0), and the other one is irreducible.

We draw the pictures illustratingthis situation (see Diagrams 3.13(a) and (b)).✲✻❅■❅■❅■❅❘❅❘❅❘❅❅❅❅❅❅❅unitaryunitarynonunitaryA++A+−A−+q −2p −2sssunitary:trivialrepresentation(a) a = 0✲✻✠✠✠❅■❅■❅■❅❘❅❘❅❘ssss❅❅❅❅❅❅❅a + q −2non-unitaryunitarynonunitaryunitaryA++A+−A−+aaa + p −2(b) a ∈Z, a > 0Diagram 3.13

DEGENERATE PRINCIPAL SERIES31Remarks. (a) It is interesting to note that both summands Sa±(X0) of Sa(X0)“see” all the barriers via their Hermitian forms, whether or not these barriers causereducibility in one summand or the other.

In fact, they see the barriers even whenthe barriers do not pass through integral points at all. (b) The unitary representation with K-types distributed along a single line whichoccurs at ˜a = 1 when p + q is even [see Diagram 3.9(a)] is particularly interesting.It is paired with the trivial representation in the local correspondence for the dualpair (O(p, q), SL(2, R)).

It is the smallest possible nontrivial unitary representationof O(p, q) and may be regarded as a “quantization of the minimal nilpotent orbit”in the argot of geometric quantization [EPW]. It turns up in physics in connectionwith the quantum Coulomb problem [Ab, AFR, DGN, Fr, On].

(c) The representations Sa+(X0) are “spherical” representations, meaning thatthey contain the trivial representation of the maximal compact subgroup, and theSa−(X0) are nonspherical representations. Notice that, for each of these families or“series” of representations, the only point on the unitary axis that can be reducibleis the point on the real axis, namely, ˜a = 0.

When p + q is even, exactly one of thetwo series is reducible at ˜a = 0, while when p+q is odd neither series is. The groupSL(2, R) has two principal series of representations, analogous to the Sa±(X0): onespherical series and one nonspherical.

The spherical series is irreducible on the realaxis, and the nonspherical series is reducible. Also, as was well known and hasbeen shown again here, the unitary spherical principal series for O(n, 1) is alwaysirreducible.

These examples perhaps created some expectation that spherical serieswere more likely to be irreducible than other series. In our example, however, whenp + q is even, either the spherical or the nonspherical series may be reducible at˜a = 0, according to the parity of (p −q)/2, while when p + q is odd, neither seriesis reducible.We note also that when a is a positive integer, the trivial K-type is alwayscontained in the central strip where nonunitarity reigns.

Thus outside the unitaryaxis and its immediate neighborhood (the complementary series), there are onlyfinitely many unitary spherical representations.Degenerate cases p ≥q = 1. In closing, we record the analogous pictures for O(p, 1).sa = 0❄unitary axisRe a = −( p−12 )✻a ∈(−(p −1), 0)Diagram 3.14.

Parameter values for unitarity of the full modules Sa±(X0).

32R. E. HOWE AND E. C. TANisunitaryunitary:trivial representation(a) a = 0iss nonunitaryunitary(b) a ∈Z, a > 0Diagram 3.15.

Unitarity at points of reducibility.The reason that the unitary representations can live so far offthe unitary axis isthat since the K-types now live along a single line, the A+−and A−+ barriers nolonger affect unitarity—only the barriers which cut offfinite-dimensional represen-tations, and these do not come into play until the finite-dimensional representationsbegin appearing at a = 0.4. The U(p, q)-module structure of homogeneous functionson light cones4.1.

The U(p, q) modules Sα,β(X0). We now wish to adapt the analysis givenfor O(p, q) in §§2 and 3 to the indefinite unitary groups U(p, q).Let Cp+q = Cp ⊕Cq denote complex (p + q)-dimensional space, parametrized bytuples (z, w), with z = (z1, z2, .

. .

, zp) in Cp and w = (w1, w2, . .

. , wq) in Cq.

Weendow Cp+q with the Hermitian form((z, w), (z′, w′))p,q = (z, z′)p −(w, w′)q =pXj=1zjz′j −qXj=1wjw′jof signature (p, q). Let U(p, q) ⊂GL(p + q, C) denote the group of linear isometriesof the form (·, ·)p,q.

Observe that the subgroup U(p) × U(q), which preserves thez variables and the w variables, is a compact subgroup of U(p, q). (In fact, it is amaximal compact subgroup.) LetX0 =η ∈Cp+q −{0}(η, η)p,q = 0=(z, w) ∈Cp+q −{0}(z, z)p = (w, w)q(4.1)

DEGENERATE PRINCIPAL SERIES33be the light cone or variety of null vectors. By Witt’s Theorem [Ja] or by elementarymeans, one can see that U(p, q) acts transitively on X0.

We wish to study the actionof U(p, q) on appropriate spaces of functions on X0.If we regard Cp+q as R2p+2q by means of coordinates xl, yl, wherezj = x2j−1 + ix2j,1 ≤j ≤p,wk = y2k−1 + iy2k,1 ≤k ≤q,(4.2)then the real part of the form (·, ·)p,q becomes the indefinite inner product ofsignature (2p, 2q), the light cone (4.1) is the same as the light cone for R2p+2q,and U(p, q) is identified to a subgroup of O(2p, 2q). Thus we are led to considerthe action of U(p, q) on the spaces Sa±(X0) defined by restriction of the actionof O(2p, 2q).

These spaces, however, can clearly never be irreducible for U(p, q),because U(p, q) commutes not simply with R×, the real scalar dilation operatorsused to define the Sa±(X0), but with C×, the complex scalar dilations. (Indeed,U(p, q) may be characterized as the subgroup of O(2p, 2q), which commutes withC×.) Thus it is appropriate to break up the spaces Sa(X0) into subspaces that arehomogeneous for the action of C× (i.e., eigenspaces for C×).

We observeC× ≃R×+ · T = {|t| · t/|t| t ∈C×}. (4.3)Here T = {s ∈C× | |s| = 1} is the usual unit circle.

Thus a character ψ of C× hasthe formψ(t) = |t|a(t/|t|)n,a ∈C×, n ∈Z. (4.4)Since |t| = (t¯t)1/2 where ¯t is the complex conjugate of t, we can formally writeψ(t) = ψα,β(t) = tα¯tβ,(4.5)with α = (a + n)/2 and β = (a −n)/2 or a = α + β, and n = α −β.

We see that apair (α, β) can occur in (4.5) if and only if α−β is an integer. For all such pairs, wedefine Sα,β(X0) ⊆Sα+β(X0) to be the ψα,β eigenspace for C×.

We clearly have adecompositionSa(X0) =Xα+β=aα−β∈ZSα,β(X0). (4.6)Further each summand in equation (4.6) is obviously invariant under the groupU(p,q).

We will study the representations of U(p,q) defined by the Sα,β(X0). Weremark that they can be interpreted as induced representations, again “degenerateprincipal series”, just as the Sa±(X0) were interpreted as induced representationsof O(2p, 2q) (see formulas (2.9)–(2.11)).

When n = 0 (or when α = β and onlythen), the representations Sα,β(X0) will be spherical, that is, will contain a vectorinvariant under the maximal compact subgroup U(p)×U(q) ⊆U(p, q). When q = 1and α = β, we have the spherical principal series of U(p, 1).To understand the U(p, q)-module structure of Sα,β(X0), we proceed as we didfor the Sa(X0): First we describe the action of the maximal compact subgroupU(p) × U(q) and then we see how the noncompact part of the Lie algebra u(p, q) ofU(p, q) moves around the U(p) × U(q)-isotypic subspaces.

34R. E. HOWE AND E. C. TAN4.2.

K-Structure of Sα,β(X0). Under the inclusion of U(p, q) in O(2p, 2q), it isclear that U(p)× U(q) ⊆O(2p)× O(2q).

Since the O(2p)× O(2q)-module structureof the Sa(X0) is described in terms of spherical harmonics (see formulas (2.12)–(2.16)), we are led to consider the structure of the space Hm(Cp) ≃Hm(R2p) as aU(p) module.Consider the algebra PR(Cp) ≃P(R2p) of polynomials on Cp considered as areal vector space. We may choose the complex coordinates z1, z2, .

. .

, zp and theircomplex conjugates z1, z2, . .

. , zp as generators for PR(Cp).

For integers α, β, letPα,β(Cp) be the space of polynomials that are homogeneous of degree α in the zjand homogeneous of degree β in the zj. This notation is consistent with that of§4.1, in the sense that Pα,β(Cp) is the ψα,β eigenspace for the action of C× onPR(Cp) by scalar dilations.

Clearly we have a decompositionPmR (Cp) ≃Xα+β=mPα,β(Cp)(4.7)of R-homogeneous polynomials of degree m into bihomogeneous polynomials ofappropriate bi-degrees.The Laplacian on Cp = R2p can be written∆=2pXj=1∂2∂x2j= 4pXj=1∂2∂zj∂zj.From this formula it is clear that ∆maps Pα,β(Cp) to Pα−1,β−1(Cp). It followsthat the harmonic polynomials have a decomposition analogous to (4.7), namely,Hm(Cp) =Xα+β=mHm(Cp) ∩Pα,β(Cp) =Xα+β=mHα,β(Cp),(4.8)where the second equation serves to define the Hα,β(Cp).Since U(p) ⊆O(2p)commutes with C×, the spaces Hα,β(Cp) will be invariant under the action of U(p)on Hm(Cp).

Hence they define representations of U(p). It is known (see [Zh]; this isalso a special case of the general results of [Ho3]) that the Hα,β(Cp) are irreducibleand mutually inequivalent U(p) modules.

We note that, in particular, the spacesHα,β(Cp) are eigenspaces for the group C× ∩U(n) ≃T of scalar unitary matrices;the eigencharacter by which T acts on Hα,β(Cp) isχα−β : z →zα−β. (4.9)If p ≥2, then Hα,β(Cp) is nonzero for all α, β ≥0; however, when p = 1, onlyHα,0(C) = Czα and H0,β(C) = Czβ are nonzero.

This degeneracy in the Hα,β(Cp)when p = 1 leads to some interesting phenomena, which will be detailed later inthe paper (see §4.5).Next consider the action of U(p) × U(q) on the algebra PR(Cp+q). Taking thetensor product of equation (4.8) with its analog for Cq gives us a decompositionHm(Cp) ⊗Hn(Cq) ≃Xm1+m2=mn1+n2=nHm1,m2(Cp) ⊗Hn1,n2(Cq).

(4.10)

DEGENERATE PRINCIPAL SERIES35The action of the scalar unitary matrices on the summand Hm1,m2(Cp)⊗Hn1,n2(Cq)is via the character χr where r = m1 −m2 + n1 −n2 [see definition (4.9)]. Usingthis, we can easily see that the maps ja,m,n of formula (2.15) split into mapsjα,β,m1,m2,n1,n2 = jα,β : Hm1,m2(Cp) ⊗Hn1,n2(Cq) →Sα,β(X0),(4.11)whereα + β = a,α −β = m1 −m2 + n1 −n2.In this connection we observe thatr22p =2pXj=1x2j =pXj=1zpzp(4.12)so that r22p is in P1,1(Cp) and multiplication by any power of r22p increases bothcomponents of bidegree equally and so does not affect their difference.Combining formulas (2.16), (4.6), (4.10), and (4.11) yields the decompositionSα,β(X0) =Xm,n≥0jα+β(Hm(Cp) ⊗Hn(Cq)) ∩Sα,β(X0)=Xm,n≥0Xm1,m2,n1,n2≥0m1+m2=mn1+n2=nm1−m2+n1−n2=α−βjα,β(Hm1,m2(Cp) ⊗Hn1,n2(Cq))=Xm1,m2,n1,n2≥0m1−m2+n1−n2=α−βjα,β(Hm1,m2(Cp) ⊗Hn1,n2(Cq))(4.13)of Sα,β(X0) into irreducible U(p) × U(q) modules.From the final form of the decomposition, we see that the U(p) × U(q) compo-nents of Sα,β(X0) form a three-parameter slice out of the full four-parameter familyof U(p) × U(q) representations that appear in PR(Cp+q).

We may refer to theseirreducible U(p) × U(q) modules as “K-types”. The middle form of the decomposi-tion emphasizes the relation of these U(p) × U(q) modules and the O(2p) × O(2q)modules of the first form of the decomposition.

We see from formula (4.10) thatthe U(p) × U(q) submodules in Hm(Cp) × Hn(Cq) can be parametrized by the pair(m1, n1), which varies in the integral points in a rectangle (see Diagram 4.14).sssssssssssssss(m1, n1) = (m, n)(m1, n1) = (0, 0)Diagram 4.14

36R. E. HOWE AND E. C. TANThe subset of points such that m1 −m2 + n1 −n2 = α −β forms a diagonal lineinside this rectangle (see Diagram 4.15).sssssssssssssss❅❅sss(m1, n1) = (m, n)(m1, n1) = (0, 0)m1 −m2 + n1 −n2 = α −βDiagram 4.15Here we should observe that sincem1 −m2 + n1 −n2 = m1 + m2 + n1 + n2 −2(m2 + n2)= m + n −2(m2 + n2),the inner sum in the middle line of formula (4.13) will be empty unless(i) m + n ≥|α −β|,(ii) m + n ≡α −β mod 2.When q = 1, there are further restrictions, described in §4.5.

For now, we assumep, q > 1.To exploit the relation between the situation under study and the results of §§2and 3, we will represent formulas (4.13) graphically as follows. We think of thepoints (m, n) for which the inner sum in the middle line of (4.13) is non-emptyas being integral points in the plane.

This gives us all points in a coset of the“diamond lattice” in the positive quadrant and lying above the line x+y = |α−β|,as illustrated by Diagram 4.16.✲x✻yrrrrrrrrrrrrrrrrrrr|α −β||α −β| + 2❅❅❅❅❅❅Diagram 4.16We further think of the “line” of U(p) × U(q) modules in the inner sum of themiddle line of (4.13) as points in a “fiber” lying over the point (m, n) in Diagram

DEGENERATE PRINCIPAL SERIES374.16. We call the region{(x, y) | x ≥0, y ≥0, x + y ≥|α −β|}the K-type region.4.3.

Action of the noncompact part of u(p, q) on Sα,β(X0). Consider nowhow the noncompact part of the Lie algebra u(p, q) acts on the U(p)× U(q) compo-nents, as described in the preceding section.

Since u(p, q) ⊆o(2p, 2q), it must actcompatibly with Diagram 2.19, so the U(p) × U(q) modules in the fiber over (m, n)must be taken to the fibers over (m ± 1, n ± 1); but since each fiber consists ofseveral U(p) × U(q) modules, we must calculate how the u(p, q) action affects thevarious points in each fiber. To formulate this we observe that the complexified Liealgebra u(p, q)C has a decompositionu(p, q)C ≃(u(p) ⊕u(q))C ⊕p+ ⊕p−,(4.17)where p± are the eigenspaces for the adjoint action of the center of u(p) ⊕u(q).Concretely we havep+ = spanSij = zi∂∂wj+ wj∂∂zi,p−= spanTij = wj∂∂zi+ zi∂∂wj.

(4.18)The following lemma is a refinement to the current situation for Lemma 2.3. Weleave the calculations justifying it to §6.2.Lemma 4.1.

There exist nonzero mapsS1001m1,m2,n1,n2,S0−101m1,m2,n1,n2,S10−10m1,m2,n1,n2,S0−1−10m1,m2,n1,n2,T 0110m1,m2,n1,n2,T −1010m1,m2,n1,n2,T 010−1m1,m2,n1,n2,T −100−1m1,m2,n1,n2,whereSε1ε2η1η2m1,m2,n1,n2 : p+ ⊗(Hm1,m2(Cp) ⊗Hn1,n2(Cq))→Hm1+ε1,m2+ε2(Cp) ⊗Hn1+η1,n2+η2(Cq),T ε1ε2η1η2m1,m2,n1,n2 : p−⊗(Hm1,m2(Cp) ⊗Hn1,n2(Cq))→Hm1+ε1,m2+ε2(Cp) ⊗Hn1+η1,n2+η2(Cq),independent of α, β, such that the action of z ∈p+ and ˜z ∈p−[see (4.18)] on theK-type jα,β(Hm1,m2(Cp) ⊗Hn1,n2(Cq)) is described by the formulaρ(z + ˜z)jα,β(φ)= A++(m, n)jα,βS1001m1,m2,n1,n2(z ⊗φ) + T 0110m1,m2,n1,n2(˜z ⊗φ)+ A−+(m, n)jα,βS0−101m1,m2,n1,n2(z ⊗φ) + T −1010m1,m2,n1,n2(˜z ⊗φ)+ A+−(m, n)jα,βS10−10m1,m2,n1,n2(z ⊗φ) + T 010−1m1,m2,n1,n2(˜z ⊗φ)+ A−−(m, n)jα,βS0−1−10m1,m2,n1,n2(z ⊗φ) + T −100−1m1,m2,n1,n2(˜z ⊗φ),

38R. E. HOWE AND E. C. TANwhere m = m1 + m2, n = n1 + n2, φ ∈Hm1,m2(Cp) ⊗Hn1,n2(Cq), and A±± =A±±2p,2q,α+β is as defined in (2.20).Proof.

See §6.2.The action described in this lemma can be pictured as follows. Imagine the fibersover (m, n) and (m + 1, n + 1) as lines of points, as suggested by Diagram 4.15.Then the effect of the S and T operators of Lemma 4.1 can be illustrated as inDiagram 4.19.✇✇✇✇✇✒❅❅❅❅❅❅❅❘✒❅❅❅❅❅❅❅❘❅❅❅❅❅❅❅■✠❅❅❅❅❅❅❅■✠(m1,m2,n1,n2)(m1+1,m2−1,n1−1,n2+1)(m1,m2+1,n1+1,n2)(m1+1,m2,n1,n2+1)(m1+2,m2−1,n1−1,n2+2)S0−1−10T 0110T −100−1S1001S0−1−10T 0110T −100−1S1001Diagram 4.19From Diagram 4.19, we see that, providing the transition coefficients A++, A−−are nonzero at (m, n) and (m + 1, n + 1) respectively, that is, providing that thefibers over (m, n) and (m + 1, n + 1) belong to the same o(2p, 2q) constituent ofSα+β(X0), application of an element of p+ followed by another from p+ moves agiven U(p) × U(q) module to its neighbor in the same fiber.

Computations in §6.2will show that by successive applications of (p+)2 [or (p−)2] we can move from anypoint in a fiber to any other. A similar analysis applies to transitions between(m, n) and (m + 1, n −1).

This implies the result: The U(p, q)-module structureof Sα,β(X0) is controlled by the O(2p, 2q)-module structure.Theorem 4.2. Assume p, q ≥2.

The image of Sα,β(X0) in any O(2p, 2q)-irreducibleconstituent of Sα+β(X0) is an irreducible U(p, q) module.In other words, the extra degree of freedom in the U(p)×U(q) fibers has no effectwhatsoever on the U(p, q) composition series of the Sα,β(X0), except in so far assome fibers may be empty. We may therefore read offthe U(p, q) constituents ofthe Sα,β(X0) from the corresponding results from O(2p, 2q) (see Diagrams 2.25–28, 2.34, and 2.35), providing we take into account Diagram 4.16 describing thenontrivial fibers for Sα,β(X0).Corollary 4.3.

Sα,β(X0) is reducible if and only if α, β ∈Z, that is, if and onlyif α + β ∈Z and α + β ≡α −β mod 2.

DEGENERATE PRINCIPAL SERIES39Proof. For reducibility, we need one of the transition coefficients (2.20) to vanishfor some (m, n) corresponding to a nonempty fiber of U(p) × U(q) modules.

Forexample, the coefficient A+−2p,2q,α+β(m, n) [see definition (2.20)] will vanish if andonly ifα + β = m −n −2(q −1).Since we know α −β ∈Z, and m −n ≡α −β mod 2, we see that α + β ∈Z andα + β ≡α −β mod 2. This implies α, β ∈Z.

Conversely, if α, β ∈Z, then we canreverse the process to find (m, n) such that A+−(m, n) = 0, implying reducibility.The same conclusion follows by considering A−+, and A++ and A−−also cannotcause reducibility unless α, β ∈Z.To describe what happens when α and β are integers, we simply have to combinethe relevant diagrams from §2 with Diagram 4.16. The various possibilities aredescribed in the following series of diagrams.

(i) α, β ∈Z+.When α, β are both nonnegative integers, the barriers A+−, A−+,and A++ all intersect the K-type region and cause Sα,β(X0) to break into fourparts: one finite-dimensional subrepresentation, two quotients, and one constituentthat is neither a quotient nor a subrepresentation (see Diagram 4.20).✲✻✲✻❅❘❅❘✠✠❅■❅■❅❅❅❅❅❅❅❅❅ttttttA++A−+A+−|α −β|α + β + 2q −2|α −β|α + βα + β + 2p −2α ∈Z+, β ∈Z+Diagram 4.20. α ∈Z+, β ∈Z+. (ii) α or β is negative, but α + β ≥1 −p −q.Here the A++ barrier nolonger intersects the region of K-types, and there are at most three constituents,all infinite dimensional.

As α + β decreases while α −β remains fixed, the barriersA+−and A−+ approach each other. When α and β are nonnegative, the pointwhere the A+−barrier leaves the K-type region, that is, the intersection of theA+−barrier with the boundary of the K-type region, is on the x-axis.

Similarly,the barrier A−+ passes through the boundary on the y-axis. As α + β decreases,A+−moves left and A−+ moves right.When α + β becomes sufficiently small(precisely, when min(α, β) ≤1 −q), the barrier A+−leaves the K-type regionthrough the antidiagonal segment {x + y = |α −β| | x ≥0, y ≥0}.

As α + β

40R. E. HOWE AND E. C. TANcontinues to decrease, the point of exit of A+−moves up the antidiagonal segmentand finally (when max(α, β) ≤1−q) moves onto the y-axis.

The barrier A−+ movesin the opposite direction. Its point of exit crosses from the y-axis to the antidiagonalsegment when min(α, β) = 1 −p and further crosses from the antidiagonal to thex-axis when max(α, β) = 1 −p.

The two barriers meet when α + β = 2 −p −q;their common point of exit from the K-type region is on the antidiagonal segmentif and only if |p −q| ≤|α −β|. The real point on the unitary axis (see §4.4) is atα+β = 1−p−q.

For a given value of α−β, exactly one of the values 2−p−q and1−p−q for α+β will be a point of reducibility. If the unitary point α+β = 1−p−qis reducible, then Sα,β(X0) breaks into a sum of two irreducible representations, asillustrated in Diagram 2.26.

An illustrative situation among the several possibilitiesdetailed above is pictured in Diagram 4.21.✲✻✲❅❘❅❘❅■❅■❅■❅❅❅❅❅❅tttA−+A+−xy|α −β||α −β|α + β + 2p −2Diagram 4.21. 1 −p ≤min(α, β) ≤1 −q.

(iii) α + β < 1 −p −q.In analogy with the remark following Lemma 3.1, themodules Sα,β(X0) and Sα′,β′(X0), where α + α′ = 1 −p−q = β + β′, are naturallydual, so the structure of Sα,β(X0) when α + β < 1 −p −q is deducible from thatof Sα′,β′(X0) as described above.4.4. Unitarity.

The unitarity criteria (3.6) for O(2p, 2q) apply directly here andthe results are as summarized in Diagrams 3.7 and 3.13. The unitary axis is definedby Re(α+β) = 1−p−q.

The possibilities for the values of α+β where the full moduleSα,β(X0) is unitary are as illustrated in Diagram 3.8 (note that ˜a = α+β+p+q−1,ε0 is + if p−q is odd, and ε1 is −if p−q is even). At a typical point of reducibility,as in Diagrams 4.20 and 4.21, the one or two constituents in the central strip arenot unitary, while the constituents outside the barriers are unitary.

Exceptions inthe range Re(α+β) ≥1−p−q occur when α = β = 0, when the finite-dimensionalsubrepresentation is the trivial representation, and when α + β = 2 −p −q, withα, β ∈Z, in which case all three components, one of which has K-types only alongthe line y −x = p −q, are unitary. See Diagram 3.9(a).4.5.

The case of U(p, 1), p > 1. For U(p, 1), degeneration of the K-types leadsto a situation substantially more complex than what is pictured in Diagram 4.16.

DEGENERATE PRINCIPAL SERIES41Since the module structure and unitarity result from an interaction between theK-types and the barriers, this change in K-type structure causes a correspondingchange in the module structure and unitarity.As noted in the discussion following formula (4.9), the only Ha,b(C) that arenonzero are Hn,0(C) and H0,n(C), n ∈Z+.Hence, instead of what is repre-sented in Diagram 4.14 as a rectangle of K-types, we have only the K-typescorresponding to the right and left sides of this rectangle, and in a given spaceSα,β(X0) ∩jα+β (Hm(Cp) ⊗Hn(C)), we will have at most two irreducible compo-nents for U(p) × U(1).This leads us to the following construction. Let us writeHn(C) =(Hn,0(C),n ≥0,H0,−n(C),n ≤0.

(4.22)Further let us represent jα+β (Hm1,m2(Cp) ⊗Hn(C)) by the point (m1 + m2, n) inR2. Here m1, m2 ∈Z+ and n ∈Z.

Sincejα+β (Hm1,m2(Cp) ⊗Hn(C)) ⊆Sα,β(X0),whereα −β = m1 −m2 + n,(4.23)and the triple (m1, m2, n) is determined by the pair (m1 + m2, n) together withα −β, we see that each point (m1 + m2, n) represents at most one K-typeinSα,β(X0). Furthermore, if the point (m, n), m ∈Z+, n ∈Z, does represent aK-type of O(2p, 2q), then we must havem1 = 12(m −(n −(α −β))) ≥0,m2 = 12(m + (n −(α −β))) ≥0, orm ≥|n −(α −β)|.

(4.24)Conversely, as long as p > 1, this condition plus m + n ≡α −β mod 2 guaranteeswe can find m1, m2, and n so that jα+β (Hm1,m2(Cp) ⊗Hn(C)) ⊆Sα,β(X0) sitsover (m, n). Thus the K-type region for Sα,β(X0) is as depicted in Diagram 4.25.

42R. E. HOWE AND E. C. TAN✲✻yx❅❅❅❅❅❅❅❅❅✲ttttttttttttttttttttttty = x + α −βα −βDiagram 4.25Note that the x-intercept is no longer |α −β| but now is just α −β.When we add barriers to this picture to determine reducibility, we find that eachof the former barriers A±,± produces two barriers, one in its original position anda second reflected through the x-axis.

Since when q = 1 the barriers A++ and A+−intersect on the x-axis, the reflection of one continues the other in a straight line.The reflection of A−+ is A−−and vice versa. Thus, with barriers inserted, Diagram4.25 is transformed into Diagram 4.26 (the shaded area is the K-type region).

DEGENERATE PRINCIPAL SERIES43✲✻❅❅❅❅❅❅❅❅❅♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❘❅❘✠❅■❅■✠✠✠❅■❅■✒✒✒A−+A++A+−A−−−(α + β + 2(p −1))−(α + β)α −βα + βα + β + 2(p −1)Diagram 4.26Study of Diagram 4.26 and similar ones yields the following conclusions when αand β are integers.Lemma 4.4. Suppose α, β ∈Z.

Consider the structure of Sα,β(X0). (a) The barrier A++ causes reducibility when β ≥0.

(b) The barrier A+−causes reducibility when α ≥0. (c) The barrier A−+ causes reducibility when β ≤−p.

(d) The barrier A−−causes reducibility when α ≤−p.Given this information, the simplest way to organize the description of the struc-ture of Sα,β(X0) is by means of a picture describing the Hasse diagram of Sα,β(X0)in various regions of the (α, β)-plane (see Diagram 4.27). The diagram is symmetricacross the line α + β = −p because of the natural duality between Sα,β(X0) andS−p−α,−p−β(X0).

44R. E. HOWE AND E. C. TAN❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅α≤−p, β≥0α+β>−p❝❝❝❅❅α + β = −p (Unitary)❄α≤−p, β≥0α+β<−p❝❝❝ ❅❅−p<α<0,β≥0❝❝α≥0, β≥0❝❝❝ ❅❅❅❅α≤−p, −p<β<0❝❝−p<α,β<0sα≥0, −p<β<0❝❝α≤−p, β≤−p❝❝❝ ❅❅❅❅−p<α<0,β≤−p❝❝α≥0, β≤−pα+β<−p❝❝❝ ❅❅α≥0, β≤−p,α+β>−p❝❝❝❅❅α + β = −p(Unitary)✻Legend:•irreducibility and unitary□finite-dimensional constituentDiagram 4.27.

Hasse diagrams for Sα,β(X0), α, β in Z.Unitarity is determined again by the conditions (3.4). The main novel feature ofthe case at hand is that some of the conditions are now vacuous because there areno K-types corresponding to transitions that would prevent unitarity.

Thus theunitary axis occurs at α + β = −p. If also |α −β| < p, that is, if α and β are inthe central square of Diagram 4.27, then all barriers miss the K-type region, andif α + β varies while α −β is fixed, the barriers continue to miss the whole K-type

DEGENERATE PRINCIPAL SERIES45region in the interval −2p + |α −β| < α + β < −|α −β|.Consequently, the representation remains irreducible and unitary for α + β inthis range. In other words, when |α −β| < p, the unitary point α + β = −p isirreducible regardless of the parity of |α −β| and the complementary series extendsto a distance p −|α −β| on either side of the unitary axis:✛✲complementaryseries−2p + |α −β|−|α −β|0sssunitary axis: Re(α + β) = −pDiagram 4.28.

Complementary series for U(p, 1)In particular when α −β = 0 [which is the case of the spherical principal seriesof U(p, 1)], the complementary series extends all the way from the unitary axis(α + β = −p) to α + β = 0, that is, α = β = 0, which contains the trivialrepresentation as its irreducible subrepresentation.This has been known sinceKostant [Ko].ss✛✲complementaryseriestrivial repn.α = −p, β = −pα = 0, β = 0unitary axis: Re(α + β) = −pDiagram 4.29. Complementary series for the spherical principalseries of U(p, 1)At points of reducibility, some of the constituents may be unitary.

When α, β ∈Z+, the “large” constituent that occupies the main part of the interior of the K-type region is unitary and the constituents living along the boundary of the K-typeregion are nonunitary (see Diagram 4.30).

46R. E. HOWE AND E. C. TAN✲✻❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅nonunitarynon-unitaryunitarynon-unitaryDiagram 4.30Exceptions arise, however, when a constituent consists only of K-types on theboundary of the K-type region.

In this case there are no transitions in the directiontransverse to the boundary, so the corresponding obstructions to unitarity do notapply and the corresponding module is unitary. In Diagram 4.31 we describe thisphenomenon for α, β ≥0.✻✲tttt❞❞tt❞❞tt❞❞t❞❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅α = 0α = 0β = 0β = 0αβα > 0, β > 0Diagram 4.31.

The filled-in circles indicate the unitary constituents.

DEGENERATE PRINCIPAL SERIES47Similar remarks apply to the other regions of (α, β) depicted in Diagram 4.27.Concluding remarks. (a) The unitary representations with K-types along a bound-ary line of the K-type region have received considerable attention in the literatureand are frequently called “ladder representations” [Ab, AFR, DGN, En, FR, SW].

(b) Consider the region of the (α, β)-plane that is shaded in Diagram 4.32.For a given value of α −β ∈Z, the complementary series consists of preciselythose values of (α, β) that fall inside the shaded region. It is well known that thefundamental group of SU(p, 1) is Z.

By considering representations of its universalcovering group, one can get a family of representations parametrized by (α, β) inwhich α −β can be an arbitrary real number. It is perhaps plausible that the setof (α, β) for which these representations are unitary is exactly the shaded region(excluding its boundary).

A. Koranyi (oral communication) has verified this forSU(2, 1).♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣♣(−p, −p)(0, −p −1)(2, −p −1)(1, −p)β = −p(0, 0)β = 0α = 0α = −p(−p, 1)(−p −2, 1)✉✉✉✉✉✉✉✉Diagram 4.32(c) When |α−β| ≥p, the K-type region contains a parallel pair of barriers (eitherA+−, A−+ or A++, A−−) when α + β = −p is the real point on the unitary axisand the complementary series is restricted to the same region as when p, q > 1 (seeDiagram 3.8). In this situation, however, we have the possibility that β ≤−p, α ≥0(or the reverse) so that at a point of reducibility, both the barriers A+−and A−+ (orboth of A++ and A−−) intersect the K-type region and create submodules.

Theseare the upper left and lower right regions in Diagram 4.27. When this happens,

48R. E. HOWE AND E. C. TANthe constituents outside the barriers are unitary, and one of them (for example, theregion above the barrier A−+ when β ≤−p ≤α+β) will have K-types contained ina diagonal strip of finite width (above A−+ and below the boundary of the K-typeregion).

Thus we obtain for U(p, 1) a family of “ladderlike” representations whoseK-types live not on a single diagonal line but in a diagonal strip. Also we shouldnote that when the real point on the unitary axis is reducible (i.e., when α −β ≡0mod 2), the two constituents are very different: one is ladderlike, with K-types ina finite strip, and the other has K-types ranging over a full quarter plane.5.

The Sp(p, q)-module structure of homogeneous functions on lightcones5.1. The Sp(p, q) modules Saj (X0).

Let H denote the algebra of quaternions,and let Hp+q ≃Hp ⊕Hq denote the vector space over H consisting of tuplesω = (u, v) = (u1, . .

. , up, v1, .

. .

, vq),ul, vl ∈H.We consider Hp+q to be endowed with the indefinite Hermitian form(ω, ω′)p,q = (u, u′)p −(v, v′)q =pXl=1u′lul −qXl=1v′lvl. (5.1)Here x denotes quaternionic conjugate of x ∈H.

Also, since H is noncommutativewe should specify that we consider Hp+q as a right vector space over H. Specifically,quaternionic scalar multiplication multiplies the coordinate of ω ∈H on the right:ωh = (u1h, u2h, . .

. , uph, v1h, .

. .

, vqh),h ∈H.This is the reason we write u′lul rather than ulu′l in formula (5.1).With thisconvention, the algebra End(Hp+q) may be realized as the space of (p + q) × (p +q) matrices with entries in H, acting on ω ∈Hp+q by the usual rules of matrixmultiplication.LetX0 =ω ∈Hp+q −{0}(ω, ω)p,q = 0(5.2)be the light cone in Hp+q. If i, j and k = ij denote the standard quaternion units,we can writeul = x4l−3 + ix4l−2 + j(x4l−1 + ix4l) = z2l−1 + jz2landvl = y4l−3 + iy4l−2 + j(y4l−1 + iy4l) = w2l−1 + jw2l.We may use the xk and yk to identify Hp+q with R4p+4q or the zk and wk to identifyHp+q with C2p+2q.

Under these identifications the light cone X0 is exactly the samelight cone as previously defined for R4p+4q or for C2p+2q.Let Sp(p, q) ⊂GL(p+q, H) be the isometry group of the Hermitian form (·, ·)p,q.The identifications of the previous paragraph give us embeddingsSp(p, q) ⊂U(2p, 2q) ⊂O(4p, 4q).

DEGENERATE PRINCIPAL SERIES49In particular, the group Sp(p, q) preserves the light cone X0. In fact, it acts tran-sitively on X0 [Ja].We can deduce in the usual fashion an action of Sp(p, q) on functions on X0,that is,(g · f)(ω) = f(g−1ω),g ∈Sp(p, q), ω ∈Hp+q.

(5.3)The quaternionic scalar dilations H× also acts on X0, and on functions on X0, andthis action of H× commutes with the action of Sp(p, q), so we can use H× to defineinvariant subspaces for Sp(p, q).Since H× is not commutative, we must consider H×-isotypic spaces instead oflooking at only eigenspaces. Specifically, write H× ≃R×+ · H1 where H1 ≃Sp(1) ≃SU(2) is the group of quaternions of norm 1.

For each integer j ≥0, there is aunique (up to equivalence) irreducible representation σj of H1 of dimension j + 1.Realize σj on a space Yj. For a ∈C, define a representation σaj of H× on Yj byσaj (h) = |h|aσj(h/|h|),h ∈H×,where |h| denotes the quaternionic norm of h. The representation σj of H1 is self-dual.

Consequently, there is a nondegenerate bilinear form ⟨·, ·⟩j (symmetric if j iseven, skew-symmetric if j is odd) satisfying⟨σaj (h)y, y′⟩j = ⟨y, σaj (h)y′⟩j,y, y′ ∈Yj, h ∈H.Define a space Saj (X0; Yj) of smooth Yj-valued functions on X0 by the recipeSaj (X0; Yj) =f ∈C∞(X0; Yj) f(ωh) = σaj (h)f(ω), h ∈H×, ω ∈X0.Define also a mapβ : Saj (X0; Yj) ⊗Yj −→C∞(X0)byβ(f ⊗y)(ω) = ⟨f(ω), y⟩j.We may define actions of Sp(p, q) on Saj (X0; Yj) ⊂C∞(X0; Yj) and on C∞(X0)by means of formula (5.3). It is trivial to check that β intertwines these actions.Furthermore, we may computeβ(f ⊗y)(ωh) = ⟨f(ωh), y⟩j= ⟨σaj (h)f(ω), y⟩j= ⟨f(ω), σaj (h)y⟩j= β(f ⊗σaj (h)(y))(ω).

(5.4)Let us denote the image of β by Saj (X0). It is not difficult to check that β is a linearisomorphism from Saj (X0; Yj) ⊗Yj to Saj (X0).

Formula (5.4) therefore shows thatthe joint action of Sp(p, q) × H× on Saj (X0) is isomorphic to the tensor product ofthe action of formula (5.3) of Sp(p, q) on Saj (X0; Yj) with the action of σaj of H×on Yj.

50R. E. HOWE AND E. C. TANFormula (5.4) implies that Saj (X0) ⊂Sa(X0).

Here Sa(X0) is as in formula(2.7). Standard arguments from harmonic analysis on compact groups tell us thatSa(X0) =Xj≥0Saj (X0),the sum being in the sense of topological vector spaces.

We should also note thatSa+(X0) =Xj≥0j evenSaj (X0)andSa−(X0) =Xj≥0j oddSaj (X0).We call Saj (X0) the σaj -isotypic component of C∞(X0). We want to describe theSaj (X0) as modules for Sp(p, q).

According to our construction of Saj (X0), this isessentially equivalent to describing Saj (X0; Yj) as an Sp(p, q) module.Our procedure for analyzing the Sp(p, q)-module structure of Saj (X0; Yj) willfollow the pattern used before in dealing with O(p, q) and U(p, q). Indeed, sinceSp(p, q) may be regarded as a subgroup of U(2p, 2q) and O(4p, 4q), much of whatwe need is already done.We only give here the additional arguments that arerequired.

In §5.2 we analyze the K-type structure, in §5.3 we discuss compositionseries and unitarity in the case p, q > 1, and in §5.4 we discuss the groups Sp(p, 1),which are particularly interesting because of degeneracy in the K-types similar tothat already described for U(p, 1) in §4.5.Before beginning our detailed description of Saj (X0; Yj), we note that they canbe thought of as induced representations, as for the representations of U(p, q) andO(p, q) previously studied. Precisely, fix a point ω0 ∈X0 and let Q be the stabilizerin Sp(p, q) of ω0.

Let P be the stabilizer of the quaternionic line ω0H through ω0.Define a map ˜α : P −→H× by the formula pω0 = ωo˜α(p). The map ˜α is, in fact,a group isomorphism ˜α :P/Q →H×.

Given any representation σaj of H×, thecomposition σaj ◦˜α defines a representation of P. It is straightforward to checkthat the space Saj (X0; Yj) is isomorphic as an Sp(p, q) module to the representationinduced from σ−aj◦˜α of P.Remark. When p = q = 1, the above representations constitute all principal seriesrepresentations, so our analysis gives a complete account of the principal series andof the unitary dual of Sp(1, 1).

Since Sp(1, 1) ≃Spin(4, 1) is the two-fold cover ofthe identity component of O(4, 1), this case reproduces the calculations of Dixmier[Di].5.2. K-structure of Saj (X0).

To investigate the structure of Saj (X0) as Sp(p, q)module, we first consider its K-structure.Here K will designate the maximalcompact subgroup Sp(p) × Sp(q) of Sp(p, q) consisting of elements of Sp(p, q) thatstabilize the decomposition Hp+q ≃Hp⊕Hq. Our experience in §§2–4 suggests thatwe try to extract an Sp(p) × H1 decomposition of Hm(R4p).

We start with somenotation.

DEGENERATE PRINCIPAL SERIES51Let H(Hp) = {f ∈P(Hp) | ∆f = 0}. Here ∆is the Laplacian with respect tothe real coordinate system defined after formula (5.2).

We observe that H(Hp) isprecisely the U(2p) harmonics in P(C2p) and is also the O(4p) harmonics in P(R4p).It is well known (see [Zh]) that irreducible finite-dimensional representations ofSp(p) can be indexed (using the highest weight) by a p-tuple of integers (ξ1, . .

. , ξp)with ξ1 ≥ξ2 ≥· · · ≥ξp ≥0.

We shall write a representation of highest weight(ξ1, ξ2, 0, . .

. , 0) as V (ξ1,ξ2)p[note that the subscript indicates the group Sp(p)].

ForH1 ≃Sp(1) ≃SU(2), we denote by V k1 , the unique irreducible representation ofSU(2) of dimension k + 1.Proposition 5.1. We have the following decomposition for Hm(R4p) as Sp(p) ×H1 ≃Sp(p) × Sp(1) module:Hm(R4p)Sp(p)×H1 ≃Pξ1≥ξ2≥0ξ1+ξ2=mV (ξ1,ξ2)p⊗V ξ1−ξ21,p > 1,V m1⊗V m1 ,p = 1.

(5.5)Hence, as Sp(p) × H1 module,C∞(S4p−1) ≃(Pξ1≥ξ2≥0 V (ξ1,ξ2)p⊗V ξ1−ξ21,p > 1,Pm≥0 V m1⊗V m1 ,p = 1.(5.6)Proof. See §6.3.Assume for the moment that p, q ≥2.

Combining the decomposition (5.5) for pand for q and an application of Clebsch Gordan Formula [Zh] yieldsHm(R4p) ⊗Hn(R4q)=Xξ1≥ξ2≥0, ξ1+ξ2=mη1≥η2≥0, η1+η2=nV (ξ1,ξ2)p⊗V (η1,η2)q⊗(V ξ1−ξ21⊗V η1−η21)=X0≤j≤m+n X nV (ξ1,ξ2)p⊗V (η1,η2)qo⊗V j1 ,(5.7)where the sum of the terms in braces is overξ1 ≥ξ2 ≥0,η1 ≥η2 ≥0,ξ1 + ξ2 =m,η1 + η2 = n,ξ1 −ξ2 + η1 −η2 ≥j ≥|(ξ1 −ξ2) −(η1 −η2)|,j ≡ξ1 −ξ2 + η1 −η2mod 2. (5.8)Note that for a fixed j, there is exactly one copy of V (ξ1,ξ2)p⊗V (η1,η2)q⊗V j1 for each4-tuple (ξ1, ξ2, η1, η2) satisfying (5.8).

It is apparent now what the K-structure ofSaj (X0) is.Define, for each 4-tuple (ξ1, ξ2, η1, η2) satisfying (5.8),ia,j : V (ξ1,ξ2)p⊗V (η1,η2)q⊗V j1 →Saj (X0),ia,j(φ) = φr2αp ,(5.9)

52R. E. HOWE AND E. C. TANwhereφ ∈V (ξ1,ξ2)p⊗V (η1,η2)q⊗V j1 ⊂Hm(R4p) ⊗Hn(R4q),r2p =2pXl=1zlzl,anda = ξ1 + ξ2 + η1 + η2 + 2α.The following statement is clear from the discussion above.Lemma 5.2.

As an Sp(p) × Sp(q) × H1 module, Saj (X0) decomposes into a directsumSaj (X0) =Xia,j(V (ξ1,ξ2)p⊗V (η1,η2)q⊗V j1 ),where the sum is over the set described in (5.8) (see Diagram 5.10).For reasons that will be apparent after the description of the transition propertiesof p ⊂sp(p, q), we continue to organize K-types of Saj (X0) by means of the points(m, n) = (ξ1 +ξ2, η1 +η2) in (Z+)2. In other words, the K-types attached to (m, n)are those satisfying (5.8).

We represent the 4-tuples (ξ1, ξ2, η1, η2) satisfying (5.8)as the integer points (ξ1 −ξ2, η1 −η2) in an auxiliary rectangleRm,n = { (x, y) | 0 ≤x ≤m, 0 ≤y ≤n }.These points fill a coset of (2Z)2 in a strip along the diagonal line x = y, as picturedin Diagram 5.10. We refer to these K-types as the fiber over (m, n).✲✻❅❅❅❅❅❅tttttttttttttttjjnm(m, m −j)(n −j, n)ξ1 −ξ2η1 −η2Diagram 5.10.

Fiber of K-types over (m, n) (if j ≡m + 1 mod 2).Diagram 5.10 helps us to see that the fiber over (m, n) is nonempty if and only if(i)m + n ≡jmod 2,(ii)m + n ≥j. (5.11)Thus, when p ≥q ≥2, the K-types of Saj (X0) fall in the region depicted in Diagram5.12.

DEGENERATE PRINCIPAL SERIES53✲✻✲✻❅❅❅❅❅❅❅ssssssssssjjxysssssssDiagram 5.12. K-type region of Saj (X0) when p, q ≥2 [consistingof points (m, n) where m + n ≡j mod 2].Now consider the case p > q = 1.

The computation parallel to (5.7) isHm(R4p) ⊗Hn(R4)=Xξ1≥ξ2≥0,ξ1+ξ2=mV (ξ1,ξ2)p⊗V n1 ⊗(V ξ1−ξ21⊗V n1 )=X0≤j≤m+nX nV (ξ1,ξ2)p⊗V n1o⊗V j1 ,where the sum of the terms in braces is overn ≥0,ξ1 ≥ξ2 ≥0,ξ1 + ξ2 = m,ξ1 −ξ2 + n ≥j ≥|(ξ1 −ξ2) −n|,ξ1 −ξ2 + n ≡jmod 2.We again associate this set of K-types of Saj (X0) to (m, n) ∈(Z+)2. If we labelthese K-types in the fiber over (m, n) by ξ1 −ξ2 = l, then we obtain all l such that|j −n| ≤l ≤min(m, j + n)(5.13)and congruent to j + n modulo 2.

This is simply the set of K-types on the lineη1 −η2 = n in Diagram 5.10. So if p > q = 1, the set of points (m, n) such that theassociated fiber of K-types is nonempty can be depicted by Diagram 5.14.

54R. E. HOWE AND E. C. TAN✲✻✲❅❅❅❅❅❅❅jjy −x = jxysssssssssssssssssssssssssDiagram 5.14.

K-type region for Saj (X0) when p > q = 1 [con-sisting of points (m, n) where m + n ≡j mod 2].For the case p = q = 1, the same arguments show that the K-types of Saj (X0)are V m1⊗V n1 , wherem + n ≥j ≥|m −n|,m + n ≡jmod 2. (5.15)The K-type region is then as depicted in Diagram 5.16.✲✻❅❅❅❅❅❅❅sssssssssssssssssssssssjjxyx −y = jy −x = jDiagram 5.16.

K-type region for Saj (X0) when p = q = 1 [con-sisting of points (m, n) where m + n ≡j mod 2].Furthermore, in this case the fiber over (m, n) consists of a single K-type, viz.,V m1⊗V n1 .

DEGENERATE PRINCIPAL SERIES555.3. Composition structure and unitarity.

Since the technique involved inthe consideration of unitarity is the same as in §§3 and 4.4, we will simply statethe results alongside our discussion of the composition series structure. WriteV (ξ1,ξ2,η1,η2)a,j=ia,jV (ξ1,ξ2)p⊗V (η1,η2)q⊗V j1if p, q > 1,V (ξ1,ξ2,η)a,j=ia,jV (ξ1,ξ2)p⊗V η1 ⊗V j1if p > q = 1,V (ξ,η)a,j=ia,jV ξ1 ⊗V η1 ⊗V j1if p = q = 1,and letU (m,n)a,j=Pξ1+ξ2=mη1+η2=nV (ξ1,ξ2,η1,η2)a,j,p ≥q > 1,Pξ1+ξ2=m V (ξ1,ξ2,n)a,j,p > q = 1,be the sum of K-types of Saj (X0) living in the fiber over (m, n).

We shall discussthe cases p ≥q ≥2, leaving the interesting cases p ≥q = 1 to §5.4. Recall thefunctions A±±p,q,a from §2 [see (2.20)].

In parallel with the cases of o(p, q) and u(p, q),we let p be the space complementary to (sp(p)⊕sp(q))C in sp(p, q)C. The followinglemma is clear from the decomposition in (5.7) and Lemma 2.3.Lemma 5.3. There exist maps, nonzero whenever the target is nonzero,T ±±m,n : p ⊗U (m,n)a,j→U (m±1,n±1)a,jsuch that the action of z ∈p on a K-type is given byρ(z)φ =A++4p,4q,a(ξ1 + ξ2, η1 + η2)T ++ξ1+ξ2, η1+η2(z ⊗φ)+ A+−4p,4q,a(ξ1 + ξ2, η1 + η2)T +−ξ1+ξ2, η1+η2(z ⊗φ)+ A−+4p,4q,a(ξ1 + ξ2, η1 + η2)T −+ξ1+ξ2, η1+η2(z ⊗φ)+ A−−4p,4q,a(ξ1 + ξ2, η1 + η2)T −−ξ1+ξ2, η1+η2(z ⊗φ),where φ ∈V (ξ1,ξ2,η1,η2)a,j⊂U (ξ1+ξ2, η1+η2)a,j.So the action of sp(p, q) [⊂u(2p, 2q) ⊂o(4p, 4q)] on each fiber over (m, n) iscompatible with Diagram 2.19.

Since each fiber has several K-types, however, inorder to have a grasp on the sp-module structure we must calculate how the sp(p, q)action affects the various points in each fiber. This is the essence of the next lemma,which we will prove in §6.3.Lemma 5.4.

Consider a fiber U (m,n)a,jof K-types in Saj (X0). By successive applica-tions of the maps T −−m+1,n+1, T ++m,n, we can move from any K-type in U (m,n)a,jto anyother.

Hence, if the transition coefficients A++4p,4q,a(m, n) and A−−4p,4q,a(m+1, n+1)are both nonzero, application of appropriate elements of the enveloping algebra U(sp)allows us to move from any K-type in U (m,n)a,jto any other. Similar remarks applyto transitions from U (m,n)a,jto any of the adjacent fibers U (m±1,n±1)a,j.

56R. E. HOWE AND E. C. TANRemark.

The adjoint action of K on p makes p an irreducible K-module. The K-types that can be reached from V (ξ1,ξ2,η1,η2)a,jby one application of p are isomorphicto constituents of V ∼= p ⊗V (ξ1,ξ2,η1,η2)a,j.

Since the K-weights of p are of the form±(ei ± fj) (see §6.3 for notation), it is not difficult to see the K-types that areembeddable in V ∼are among the 16 whose highest weights are obtained by addingone of the vectors (±1, 0, ±1, 0), (±1, 0, 0, ±1), (0, ±1, ±1, 0), or (0, ±1, 0, ±1) to(ξ1, ξ2, η1, η2). We will see that by moving from U (m,n)a,jto U (m+1,n+1)a,jby means ofp and then back again we can move from V (ξ1,ξ2,η1,η2)a,jto any of its eight nearestneighbors, as illustrated in Diagram 5.17 (see also Diagram 5.10).✒❅❅❅❅❅❅❅❅■✠❅❅❅❅❅❅❅❅❘✲✻✛❄✉✉✉✉✉✉✉✉✉(0,0,1,−1)(1,−1,1,−1)(1,−1,0,0)(1,−1,−1,1)(0,0,−1,1)(−1,1,−1,1)(−1,1,0,0)(−1,1,1,−1)Diagram 5.17In this diagram, the labels on the arrows indicate increments to (ξ1, ξ2, [0]η1, η2)and the arrows show which increments can be achieved by applying T −−m+1,n+1T ++m,n.Thus the arrow to (1, −1, 0, 0) indicates that we can move from V (ξ1,ξ2,η1,η2)a,jtoV (ξ1+1,ξ2−1,η1,η2)a,jand so forth.

Similar pictures apply to the other transitions. Ofcourse, if (ξ1, ξ2, η1, η2) lies on the boundary of the K-type region (see Diagram5.10), then transitions that would move it outside the region are suppressed.Study of Diagrams 5.10 and 5.17 will convince the reader that there are nobarriers to movement between K-types in a given fiber, as long as one can moveback and forth between the fiber and an adjacent fiber, but from the geometry ofthe barriers (see Diagram 2.22) we can see that the only occasion on which onecannot pass between a fiber and an adjacent one is for U (0,0)0,0 —and this consistsonly of the trivial K-type, so there is nothing to prove in this case.

Thus by usingLemma 5.4, we can derive the following result.Theorem 5.5. (a) The image in any O(4p, 4q)-irreducible component of Sa(X0)of a given Saj (X0) is an irreducible module for Sp(p, q) × H1.In particular, ifp ≥q > 1, then Saj (X0) is reducible if and only if a ∈Z and a ≡j mod 2.

DEGENERATE PRINCIPAL SERIES57(b) Saj (X0) will have Hermitian constituents if and only if a is on the unitaryaxis (in which case Saj (X0) will be unitary) or a is real (in which case all itsconstituents will be Hermitian). A given Sp(p, q) constituent of Saj (X0) is unitaryif and only if the O(4p, 4q) transitions between the nonempty fibers of the Sp(p, q)constituent are consistent with unitarity.Remark.

As in the case of U(p, q) modules Sα,β(X0), one obtains an Sp(p, q)composition series for an Sp(1) eigenspace of Sa(X0) by intersecting it with theO(4p, 4q) composition series for Sa(X0).Proof. Again this follows essentially from Lemmas 5.3 and 5.4.

The pictures whenp ≥q ≥2 will be similar to those under Case EE in §§2 and 3 (see Diagrams 3.13,4.20, and 4.21).5.4. The case of Sp(p, 1).

The K-type regions are depicted in Diagrams 5.14 and5.16. For p > q = 1, there is a “line” of K-types over each point in the K-typeregion.

We note that unlike the situation of U(p, 1), K-types along each fiber canbe moved to adjacent ones via operators in U(sp(p, 1)). The analogs of Lemmas 5.3and 5.4 hold.

We do not state them formally. The following result is easy.Theorem 5.6.

Assume p ≥q = 1. If a ̸∈Z or a ̸≡j mod 2, then Saj (X0) isirreducible for Sp(p, 1).Following §4.5, we organize our description of the structure of Sp(p, 1), p > 1,by means of a picture (see Diagram 5.18) describing the Hasse diagrams of Saj (X0)in the (a, j)-plane (a ∈R).Observe that the diagram is symmetric about theaxis a = −2p −1, because of the duality between the modules S−2p−1+sj(X0) andS−2p−1−sj(X0) (see Lemma 3.1).

58R. E. HOWE AND E. C. TAN✲✻❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅❅sssssssssssss❝s❅❅ss❅❅ssss❅❅ssss❝ss❅❅❝s❝s❝s−2p −1unitary axisirreducibleand unitary(−2p, 2p −2)−20aja + j = −2a = −2p −1−4p −2a = −2pj = a + 4p −2a = jLegend:◦nonunitary constituent•unitary constituent■finite-dimensional constituent□trivial representationDiagram 5.18.

Submodule structure of Saj (X0) when a ≡jmod 2 for Sp(p, 1).Again the existence of the complementary series is very interesting. Notice thatin the triangular region given by a −j ≥−2 −4p and a + j ≤−2, the modulesare irreducible and unitarizable.

The length of a horizontal line in this trianglewill be twice the length of the complementary series for a fixed j, that is, thereis a complementary series of length 2p −j −1 for each j ≤2p −2. The extremecase when j = 0, that is, the spherical principal series, has complementary seriesof length 2p −1.

Unlike the case for U(2p, 1) module Sα,β(X0), where α = β, thecomplementary series does not run till a = 0. This can be explained by observingthat, for U(p, 1), the barriers A++ and A+−always intersect on the x-axis, whilefor Sp(p, 1) there is a gap of length 2 between the x-intercepts of the A++ andA+−barriers.

This means that as a increases from the unitary axis it is alwaysthe A+−barrier that first causes reducibility and prevents further unitarity, evenin the extreme case j = 0.Diagram 5.19 describes the situation for Sp(1, 1), with the same legends as inDiagram 5.18. We hope it will be self-explanatory.

DEGENERATE PRINCIPAL SERIES59✲✲✲✲✲j = 1j = 2j = 3j = 4❝❝sssssss❝❝❝❝ss❝s♣s❝❝❅❅sss❅❅ssss❝s❅❅s❝s❅❅−3−2−101aa = −2a = −3unitary axisj = aj = a + 2irred.j = a + 4Legend:◦nonunitary constituentsunitary constituent■ladder representation□finite-dimensional constituent⊡trivial representationDiagram 5.19. Submodule structure of Saj (X0) for Sp(1, 1).Remarks.

We add some remarks in the case of Sp(2, 1). Our results showed thatthe K-types of Saj (X0) are of the form V (ξ1,ξ2)2⊗V n1 where ξ1, ξ2, n satisfies (5.13)(see Diagram 5.12).

Let us instead plot the second and third coordinates, thatis, (ξ2, n), fixing ξ1; this is a section of the K-type region parallel to the ξ1 axis(see Diagram 5.20; here ξ1 (≥j) is fixed and we also have the parity conditionξ1 + ξ2 + n ∈2Z).The following statements are based on the second author’s thesis (see [Ta]), andwe refer our readers to that for full details.We will be somewhat sketchy.Ifwe consider arbitrary principal series representations of Sp(2, 1), we get highestweights (ξ1, ξ2, n) that live in regions whose convex hulls are wedges in R3. If wetake sections parallel to the ξ1-axis, we get Diagram 5.21 (note that Diagram 5.20is a “degenerate” case of this diagram).

60R. E. HOWE AND E. C. TAN✲✻nξ2❅❅❅❅❅❅❅❅❅❅ξ1 −jξ1 −j(ξ1, j)(ξ1 −j, 2j)Diagram 5.20✲✻❅❅❅❅❅❅❅❅❅❅ξ2n45o45o135o135o90o135o135oDiagram 5.21The multiplicities of K-types along the boundary of the figure is one, and themultiplicities “grow” linearly with the distance from the boundary, attaining a con-stant towards the ‘central’ region, much like the way the weight multiplicities of anirreducible sl(3) representation behave in relation to its weight diagram.

Further, ifwe look at K-type diagrams for the quotients of the principal series representations,we get diagrams like Diagram 5.22.

DEGENERATE PRINCIPAL SERIES61✻n✲ξ2❅❅❅❅❅or❅❅❅❅❅❅❅❅❅❅Diagram 5.22The multiplicities behave as described in the previous paragraph. Details are omit-ted since the results arise from painstaking computations involving Blattner’s For-mula, Branching Rules, and known composition series structure for Sp(2, 1).

Areason for the nice multiplicity result in this case may be the already apparentlinearity in the branching rules for Sp(2) × Sp(1) [a maximal compact subgroup ofSp(2, 1)] to Sp(1) × Sp(1).6. Technicalities6.1.

Computations for O(p, q). Let x1, .

. .

, xn be the standard system of coor-dinates in Rn and let∆=nXj=1∂2∂x2j,r2 =nXj=1x2j,E =nXj=1xj∂∂xj.We have the commutator[∆, r2] = 4E + 2nand so {∆, r2, 4E + 2n} span a Lie algebra isomorphic to sl(2, R). If P is a homo-geneous polynomial in P(Rn), let deg P = E(P) be the homogeneous degree of P.Then∆(r2P) = r2∆P + (4E + 2n)P= r2∆P + (4 deg P + 2n)P.(6.1)There are unique harmonic polynomials hi such thatP =lXi=1hi(r2)ai(6.2)

62R. E. HOWE AND E. C. TANfor some positive integer l and deg P = deg hi + 2ai.

To compute the hi’s, we use(6.1) inductively. If ∆P = 0, then l = 1, h1 = P, and a1 = 0.

If ∆2P = 0, compute∆(r2∆P) =r2∆2P + (4(deg P −2) + 2n)∆P⇒∆P −r2∆P4(deg P −2) + 2n= 0and getP = P ∼+ r2∆P4(deg P −2) + 2n,(6.3)where for convenience we will writeP ∼= P −r2∆P4(deg P −2) + 2n. (It should be clear from context which space Rn we are working with when thedefinition above is being used.) One could proceed inductively using (6.1) to obtainthe decomposition in (6.2) for any homogeneous P.If φ = h1 ⊗h2 ∈Hm(Rp) ⊗Hn(Rq), defineT ++ijφ = (xih1)∼(yjh2)∼,T +−ijφ = c2(xih1)∼∂h2∂yj,T −+ijφ = c1∂h1∂xi(yjh2)∼,T −−ijφ = c1c2∂h1∂xi∂h2∂yj,wherec1 =1(2 deg h1 −2 + p)andc2 =1(2 deg h2 −2 + q).Extend the above definitions linearly to all elements in Hm(Rp) ⊗Hn(Rq).Proof of Lemma 2.3.Suffices to show forz = xi∂∂yj+ yj∂∂xi∈p.Thus, for h = h1h2r2γpwhere h1 ∈Hm(Rp) and h2 ∈Hn(Rq), computexi∂∂yj+ yj∂∂xi(h)= (xih1)∂h2∂yjr2γp +∂h1∂xi(yjh2)r2γp + 2γ(xih1)(yjh2)r2(γ−1)p.From (6.3) we havexih1 = (xih1)∼+ c1∂h1∂xir2p

DEGENERATE PRINCIPAL SERIES63andyjh2 = (yjh2)∼+ c2∂h2∂yjr2q = (yjh2)∼+ c2∂h2∂yjr2psince r2p = r2q on X0. Hence,xi∂∂yj+ yj∂∂xi(h)=(xih1)∼+ c1∂h1∂xir2p ∂h2∂yjr2γp +∂h1∂xi (yjh2)∼+ c2∂h2∂yjr2pr2γp+ 2γ(xih1)∼+ c1∂h1∂xir2p (yjh2)∼+ c2∂h2∂yjr2pr2(γ−1)p,which gives the required result if p ≥q ≥2.Proof of Lemma 3.1.We will first show that if dx, dy are the rotation-invariantprobability measures on Sp−1 and Sq−1 respectively, thenJ(f) =ZSp−1×Sq−1 f(x, y) dx dyis an O(p, q)-invariant linear functional on S2−p−q(X0).

To this effect, letX0s = { (x, y) ∈X0 | 1s ≤rp(x) ≤s },s > 0.The O(p, q)-invariant measure on X0 ≃Sp−1×Sq−1×R×+ is tp+q−3 dx dy dt. Observethat for f ∈S2−p−q(X0)12 log sZX0sf(tx, ty) tp+q−3 dx dy dt =12 log s Z s1s1t dt!J(f) = J(f).If g ∈O(p, q), the image of X0s under g isgX0s =n(t′x′, t′y′) ∈X0 x′ ∈Sp−1, y′ ∈Sq−1, φ(x′, y′)s≤t′ ≤sφ(x′, y′)o,where φ(x′, y′) is a smooth function on Sp−1 × Sq−1.

Thus,J(g−1 · f) =12 log sZg·X0sf(t′x′, t′y′) t′(p+q−3) dx′ dy′ dt′=12 log sZSp−1×Sq−1 f(x′, y′) Z sφ(x′,y′)φ(x′,y′)sdt′t′!dx′ dy′=ZSp−1×Sq−1 f(x′, y′) dx′ dy′=J(f),which proves our earlier assertion.For f ∈S−a+2−p−q(X0) and h ∈Sa(X0) define⟨f, h⟩= J(fh) =ZSp−1×Sq−1 f(x, y)h(x, y) dx dy.

64R. E. HOWE AND E. C. TANThen it is easy to see that this is an O(p, q)-invariant Hermitian pairing betweenS−a+2−p−q(X0) with Sa(X0).

It gives a Hermitian inner product on Sa(X0) if−a + 2 −p −q = aorRe a = −(p + q)2+ 1.This proves Lemma 3.1.6.2. Computations for U(p, q).

Define for Sij and Tij as in (4.18) and φ =h1 ⊗h2 ∈Hm1,m2(Cp) ⊗Hn1,n2(Cq),S1001(Sij ⊗(h1 ⊗h2)) = (zih1)∼(wjh2)∼,S0−101(Sij ⊗(h1 ⊗h2)) = c1∂h1∂zi(wjh2)∼,S10−10(Sij ⊗(h1 ⊗h2)) = c2(zih1)∼∂h2∂wj,S0−1−10(Sij ⊗(h1 ⊗h2)) = c1c2∂h1∂zi∂h2∂wj,T 0110(Tij ⊗(h1 ⊗h2)) = (zih1)∼(wjh2)∼,T −1010(Tij ⊗(h1 ⊗h2)) = c1∂h1∂zi(wjh2)∼,T 010−1(Tij ⊗(h1 ⊗h2)) = c2(zih1)∼∂h2∂wj,T −100−1(Tij ⊗(h1 ⊗h2)) = c1c2∂h1∂zi∂h2∂wj(6.4)wherec1 =1(p + m1 + m2 −1),c2 =1(q + n1 + n2 −1),P ∼=P −r2n∆nPn + degn,z P + degn,¯z P −1. [Note that P ∼can be shown to be harmonic if ∆2nP = 0 through computationssimilar to those before (6.3).

Here degn,z P and degn,¯z P are the degrees in z and ¯z ofP in P(Cn).] Extend the definitions (6.4) linearly to all of Hm1,m2(Cp)⊗Hn1,n2(Cq).Lemma 4.1 then follows from a straightforward computation as in Lemma 2.3.We still need to verify that one could move freely within each fiber (see thediscussion after Diagram 4.19).

Write [see (4.11)]V α,β(m1,m2,n1,n2) = jα,β(Hm1,m2(Cp) ⊗Hn1,n2(Cq)).Let us show that there is a nonzero map U + from U(u(p, q)) withU + : V α,β(m1,m2,n1,n2) →V α,β(m1+1,m2−1,n1−1,n2+1). (6.5)

DEGENERATE PRINCIPAL SERIES65This is a movement in the south-westerly direction along the fiber depicted inDiagram 4.15. Let Π(m1,m2,n1,n2) be the projection to the K-type V α,β(m1,m2,n1,n2),that is,Π(m1,m2,n1,n2) : Sα,β(X0) →V α,β(m1,m2,n1,n2).

(6.6)This projection certainly arises from the action of K on Sα,β(X0). Letm = m1 + m2,n = n1 + n2and writeSε1,ε2,η1,η2ij(φ) =jα,β (Sε1,ε2,η1,η2(Sij ⊗φ)) ,T ε1,ε2,η1,η2ij(φ) =jα,β (T ε1,ε2,η1,η2(Tij ⊗φ))(6.7)for φ ∈Hm1,m2(Cp) ⊗Hn1,n2(Cq).

Then the maps(a) Π(ρ(Sij){Π(m1+1, m2, n1, n2+1)ρ(Sij)(jα,β(φ))})= A−−(m + 1, n + 1)A++(m, n)S0−1−10ij(S1001ij(φ)),(b) Π(ρ(Sij){Π(m1+1, m2, n1−1, n2)ρ(Sij)(jα,β(φ))}),= A−+(m + 1, n −1)A+−(m, n)S0−101ij(S10−10ij(φ)),(c) Π(ρ(Sij){Π(m1, m2−1, n1−1, n2)ρ(Sij)(jα,β(φ))})= A++(m −1, n −1)A−−(m, n)S1001ij(S0−1−10ij(φ)),(d) Π(ρ(Sij){Π(m1, m2−1, n1, n2+1)ρ(Sij)(jα,β(φ))})= A+−(m −1, n + 1)A−+(m, n)S10−10ij(S0−101ij(φ)),whereΠ = Π(m1+1, m2−1, n1−1, n2+1),have the mapping property as in (6.5). It suffices to show that at least one ofthem has nonzero coefficients.

A simple check using formulas (2.20) verifies thatall four coefficients vanish only when m = n = 0 = α = β, but when this happens,V α,β(m1±1,m2∓1,n1∓1,n2±1) vanishes, that is, the fiber at (0, 0) has only one K-type.Thus, we can always move in the south-westerly direction along the fiber. Theproof for the north-easterly movement is similar.6.3.

Computations for Sp(p, q). Choose z1, .

. .

, z2p, w1, . .

. , w2q, z1, .

. .,z2p, w1,.

. .

, w2q as a system of coordinates in (Hp+q)C. The complexified Lie algebra ofSp(p, q) may be described as the set of matrices in M2p,2q(C)A11A12A13A14A21−At11A23A24At24−At14A33A34−At23At13A43−At33where A11, A12, A21 ∈Mp(C), A33, A34, A43 ∈Mq(C), A13, A14, A23, A24 ∈Mp,q(C) with A12 = At12, A21 = At21, A34 = At34 and A43 = At43. Note that sp(p)Cand sp(q)C are embedded block diagonally in sp(p, q)C.It is not difficult to check that the following first-order differential operators spanthe action of sp(p)C on the space of polynomial functions on (Hp+q)C [1 ≤i, j ≤pfor (a) and 1 ≤i ≤j ≤p for (b) and (c)]:

66R. E. HOWE AND E. C. TAN(a)A1ij = zi∂∂zj−zj∂∂zi+ zi+p∂∂zj+p−zj+p∂∂zi+p,(b)B1ij = zi∂∂zj+p−zj+p∂∂zi+ zj∂∂zi+p−zi+p∂∂zj,(c)C1ij = zj+p∂∂zi−zi∂∂zj+p+ zi+p∂∂zj−zj∂∂zi+p.

(6.8)Similarly, the following describes the action of sp(q)C [1 ≤i, j ≤q for (a) and1 ≤i ≤j ≤q for (b) and (c)]:(a)A2ij = wi∂∂wj−wj∂∂wi+ wi+q∂∂wj+q−wj+q∂∂wi+q,(b)B2ij = wi∂∂wj+q−wj+q∂∂wi+ wj∂∂wi+q−wi+q∂∂wj,(c)C2ij = wj+q∂∂wi−wi∂∂wj+q+ wi+q∂∂wj−wj∂∂wi+q. (6.9)Finally, the space p complementary to sp(p)C ⊕sp(q)C in sp(p, q)C is spanned bythe following (1 ≤i ≤p and 1 ≤j ≤q):(a)Pij = zi∂∂wj+ wq+j∂∂zp+i+ wj∂∂zi+ zp+i∂∂wq+j,(b)Qij = zi∂∂wq+j−wj∂∂zp+i+ wq+j∂∂zi−zp+i∂∂wj,(c)Uij = zp+i∂∂wj−wq+j∂∂zi+ wj∂∂zp+i−zi∂∂wq+j,(d)Vij = zp+i∂∂wq+j+ wj∂∂zi+ wq+j∂∂zp+i+ zi∂∂wj.

(6.10)LetE11 =2pXl=1zl∂∂zl,E22 =2pXl=1zl∂∂zl,E12 =pXl=1(zl+p∂∂zl−zl∂∂zl+p),andE21 =pXl=1zl∂∂zl+p−zl+p∂∂zl.It is not difficult to check that {∆p, r2p, E11, E12, E21, E22} commutes with sp(p) onthe space of polynomial functionsP(HpC) ≃C[z1, . .

. , z2p, z1, .

. .

, z2p].They span a Lie algebra isomorphic too(4)∗C =nX ∈gl(4) X ˜J + ˜JX = 0 and XJ + JX = 0oC ,

DEGENERATE PRINCIPAL SERIES67where˜J = 0I2I20andJ =I200−I2.Here I2 is the 2 × 2 identity matrix. Observe that {E11, E12, E21, E22} spans the Liealgebra gl(2) where Eij corresponds to the standard matrix unit at (i, j) in gl(2).In particular, one sees that the Lie algebra spanned by {E11 −E22, E12, E21}corresponds to the infinitesimal action of H1 ≃Sp(1).With the notation as in §5.2, we have the following result, which implies Propo-sition 5.1 immediately.Lemma 6.1.

The decomposition (for p > 1)H(Hp)Sp(p)×H1 =XV (ξ1,ξ2)p⊗V ξ1−ξ21runs through all representations V (ξ1,ξ2)pof Sp(p).The Sp(p) module V (ξ1,ξ2)p⊗V ξ1−ξ21is generated by the joint highest weight vectorhp,(ξ1,ξ2) = zξ1−ξ21z1z1+pz2z2+pξ2. (6.11)For p = 1 the isotypic decomposition of H(H) isH(H)Sp(1)×Sp(1) =∞Xm=0V m1⊗V m1 ,where V m1⊗V m1is generated by h1,m = zm1 .Remark.

If we apply the lowering operator E21 in sp(1) to the joint highest weightvector (6.11), we find that a spanning set of Sp(p) highest weight vectors in H(Hp)are given by(hp,(ξ1,ξ2),j = zp+1jhp,(ξ1−j,ξ2),j = 0, 1, ..., ξ1 −ξ2,h1,m,j = z2jh1,m−j,j = 0, 1, ..., m.(6.12)Proof. Choose the positive roots of sp(p)C to be those described by (6.8)(a) (for1 ≤i ≤j ≤p) and (6.8)(b).

It is easy to check that hn,(ξ1,ξ2) are sp(p)C × sp(1)Chighest weight vectors of weight (ξ1, ξ2, 0, 0, . .

. , 0) ⊗(ξ2 −ξ2).

For the remainingpart of the first assertion, we refer to [Ho2].Proof of Lemma 5.4. We proceed to compute the action of the p-part of Sp(p, q) onSaj (X0).

From Lemma 6.1, the Sp(p) × Sp(q) highest weight vectors in Hm(R4p) ⊗Hn(R4q) arehp,(a1,a2),jhq,(b1,b2),k = zp+1jhp,(a1−j,a2)wq+1khq,(b1−k,b2).

68R. E. HOWE AND E. C. TANFor fixed (a1, a2) and (b1, b2), these vectors span an sp(1) module isomorphicto V a1−a21⊗V b1−b21.

The Clebsch-Gordan formula tells us that the sp(1) highestweight vectors in this module are the functionshp,(a1−j,a2)hq,(b1−j,b2)z1zp+1w1wq+1j.To adjust the homogeneity of these functions, we should multiply by an appropriatepower of r2p. Thus we wish to study the functionsγ(a1, a2, b1, b2, c, d) = za11z1zp+1z2zp+2a2wb11w1wq+1w2wq+2b2z1zq+1w1wq+1c(r2p)d.Here aj, bj, and c are positive integers and d is a complex number.

Note that ouruse here of aj, bj is slightly different from that of a few lines above. A check showsthat γ(a1, a2, b1, b2, c, d) is an Sp(p) × Sp(q) highest weight vector of Sp(p) weight(a1 + a2 + c, a2, 0, ..., 0) and Sp(q) weight (b1 + b2 + c, b2, 0, ..., 0).

Also it is ansp(1) highest weight vector of weight a1 + b1.Recall the notation in §5.3. The possible components in p⊗V (ξ1,ξ2)p⊗V (η1,η2)qarethe sixteen components obtained by adding (±1, 0, ±1, 0), (±1, 0, 0, ±1), (0, ±1, ±1, 0)and (0, ±1, 0, ±1) to (ξ1, ξ2, η1, η2).

Our intention will be to show that we can movefrom the fiber over (m, n) to a neighboring fiber (m ± 1, n ± 1) and then back, anddoing this a sufficient number of times to get to any other points in the originalfiber.Because the moves between, say (ξ1, ξ2, η1, η2) to (ξ1 + 1, ξ2, η1 + 1, η2), andin the reverse direction are essentially adjoint to one another (exactly so on theunitary axis), we can do one move if and only if we can do the reverse move. Thusit suffices to check if we can do eight of the sixteen moves.To show that we can do the (1,0,1,0) move, we apply Q11 [see (6.10)] to one ofour functions γ :Q11γ(a1, a2, b1, b2, c, d) = dγ(a1, a2, b1, b2, c + 1, d −1).

(6.13)To show that we can do the (1,0,0,1) move, consider(A221Q11 −(b1 + c + 1)Q12)γ(a1, a2, b1, b2, c, d)= −b1dγ(a1 + 1, a2, b1 −1, b2 + 1, c, d −1). (6.14)Similarly for the (0,1,1,0) move, we appeal to the formula(A121Q11 −(a1 + c + 1)Q21)γ(a1, a2, b1, b2, c, d)= −a1dγ(a1 −1, a2 + 1, b1 + 1, b2, c, d −1).

(6.15)Finally, for the (0,1,0,1) move, we can verify that the operatorA121A221Q11 −(a1 + c + 1)A221Q21−(b1 + c + 1)A121Q12 + (a1 + c + 1)(b1 + c + 1)Q22applied to γ(a1, a2, b1, b2, c, d) yieldsc(a1 + b1 + c + 1)dγ(a1, a2 + 1, b1, b2 + 1, c −1, d −1). (6.16)

DEGENERATE PRINCIPAL SERIES69Thus in passing from the fiber U (m,n)a,jto the fiber U (m+1,n+1)a,j(see §5.3, beforeLemma 5.3) by means of the action of the Qij, we can add all four of the conceivablevectors (1, 0, 1, 0), (0, 1, 1, 0), (1, 0, 0, 1), and (0, 1, 0, 1) to a given highest weight ofU (m,n)a,j. By taking adjoints, we can add the negatives of these vectors when passingfrom U (m+1,n+1)a,jto U (m,n)a,j.

Thus passing from U (m,n)a,jto U (m+1,n+1)a,jand backagain allows us to move from a given K-type in U (m,n)a,jto any of its eight nearestneighbors, as illustrated in Diagram 5.17.If the vanishing of either of the transition coefficients A++4p,4q,a(m, n) or A−−4p,4q,a(m+1, n+ 1) prevents us from moving to U (m+1,n+1)a,jor back to U (m,n)a,j, we can equallywell move from U (m,n)a,jdown to U (m−1,n−1)a,jand back up to achieve the same end.Study of the diagrams of the K-type regions (Diagrams 5.12, 5.14, 5.16) and thepositions of the barriers (Diagram 2.34) shows that the only time neither of thesestrategies works is when (m, n) is on the border of the K-type region; but in thiscase, we can see from Diagram 5.10 that the fiber U (m,n)a,jcontains only one K-type,so there is nothing to show. Although we do not need them, similar calculations maybe done for the transitions from U (m,n)a,jto U (m+1,n−1)a,jand back.

This concludesthe proof of Lemma 5.4.Concluding remark. Although in this paper we have been exclusively concernedwith questions of infinitesimal structure, our results have some implications forharmonic analysis on hyperbolic spaces (also describable as isotropic (reductive)symmetric spaces [Sc]).

To describe these, consider the complex projective hyper-boloid Y = U(p, q)/(U(p −1, q) × U(1)). Similar remarks apply to the quaternionichyperboloid.

As noted in [Sc], the real hyperboloidX1 = {w ∈R2p+2q | (w, w)p,q = 1}fibers over Y with fiber U(1). In fact, this is a principle U(1)-bundle.

It is wellknown (see the references on the first page, especially [Sc]) that all the representa-tions occurring in the harmonic analysis of the O(2p, 2q) action on X1 are amongthose considered in §§2 and 3. Our results imply that the restriction of any one ofthese representations to U(p, q) decomposes discretely; more precisely, for any ofthese representations of O(2p, 2q), the eigenspaces of the fiber U(1) of X1 over Ywill be irreducible modules for U(p, q).

It follows that the U(p, q) harmonic analysisof any line bundle constructed from the fibration X1 −→Y reduces immediatelyto the O(2p, 2q) harmonic analysis on X1. In particular, harmonic analysis on Yreduces to harmonic analysis on X1.7.

Afterword of cautionIn this article we have tried to exhibit some of the elegance of representationtheory of semisimple groups. We hope some readers will be inspired to go further,in the texts [Kn], [Wa], or the expository accounts [Ho4], [Vo4].

Here at the end,we should warn the reader otherwise unfamiliar with representation theory thatthe examples we have studied are still only a small, and in some ways misleadinglycomprehensible, sample of the full spectrum of representations of semisimple groups.One way to locate our representations in relation to the whole is to recall theirdescription as degenerate principal series [see, e.g., formulas (2.9)–(2.11)]. They are

70R. E. HOWE AND E. C. TANrepresentations induced from certain finite-dimensional representations of the groupP1, the stabilizer of an isotropic line.

This group is a maximal parabolic subgroup,and it is its maximality, and the corresponding smallness of the coset space G/P1,that leads to the use of the adjective “ degenerate”. If one considers representationsinduced from finite-dimensional irreducible representations of a minimal, ratherthan maximal, parabolic subgroup, then one has the (ordinary or nondegenerate)principal series.

[For O(p, q) a general parabolic subgroup is describable as thestabilizer of a nested sequence X1 ⊂X2 ⊂X3 ⊂· · · ⊂Xk of isotropic subspaces. Ifthe sequence contains a subspace of every dimension from one to min(p,q), then theparabolic stabilizing it is minimal.

Similar remarks apply to U(p, q) and Sp(p, q). ]Principal series have the following properties:(1) They come in continuous families, parametrized by several (in place of thesingle parameter of our examples) complex variables.

(2) For generic values of the parameters, they are irreducible. (3) In general, they have finite composition series.

(4) Any (reasonable) irreducible representation is equivalent (via an appropri-ate notion of equivalence) to a constituent of some principal series.Properties (1)–(3) are qualitatively similar to what we saw in this paper, and prop-erty (4) tells us that if we understand the principal series, then we understandan essential part, in particular the building blocks, of semisimple representationtheory.It might seem that properties (1)–(4) virtually finish the subject, but no onehas so far managed to achieve control of general principal series comparable to thepicture given for the examples in this paper. The point is that the combinatorialstructure of the composition series of a general principal series, and of its con-stituents, is rather spectacularly more complex than what we encountered in ourexamples.

(See [CC] for a recent discussion.) Some of the most important advancesin the subject (Langlands classification [Kn, Wa, Vo2]; Kazhdan-Lusztig conjec-tures [BB, BK, KL, Vo3]) have involved obtaining significant information aboutthe composition series of principal series.Let us remark that when q = 1, our maximal parabolic is the minimal andonly proper parabolic subgroup, and in this case our representations are ordinary,not degenerate, principal series.

When p is also small, our representations in factinclude all, or essentially all, principal series, so that we find all the irreduciblerepresentations, and since we know which are unitary, we classify the unitary dual.Specifically we obtain all principal series for U(1, 1) and for Sp(1, 1) (≃Spin(4, 1)),and we obtain all principal series up to twisting by a quasi-character of the wholegroup for O(2, 1) and U(2, 1).Since, at least when p and q are greater than one, the representations we haveconsidered are quite special, one may ask if any interest attaches to them, aside fromthe inherent attractiveness of the picture we have tried to present. In fact, theserepresentations arise in studying harmonic analysis on hyperboloids.

The full im-port of the term “harmonic analysis” has never been completely decided, but manywould agree that it certainly includes the decomposition of L2-spaces of homoge-neous spaces. Among the homogeneous spaces of semisimple Lie groups, the classof “semisimple symmetric spaces” or “affine symmetric spaces” [F-J] recommends

DEGENERATE PRINCIPAL SERIES71itself as particularly interesting. The hyperboloidsXt = {w ∈Rp+q | (w, w)p,q = t}are very closely related to (precisely, are principal fibre bundles with compact fibreover) some of the simplest examples of semisimple symmetric spaces [Sc], and therepresentations we have studied enable one to decompose the L2-spaces of hyper-boloids.

The connection with harmonic analysis on semisimple symmetric spaceswas the motivation for many of the papers cited in the introduction.References[Ab]H. Abarbanel, The inverse r-squared force; an introduction to its symmetries, Essays inHonor of Valentine Bargmann (E. Lieb, B. Simon, and A. Wightman, eds. ), PrincetonUniv.

Press, Princeton, NJ, 1976.[AFR]R. Anderson, J. Fischer, and R. Raczka, Coupling problem for U(p, q) ladder represen-tations.

I, Proc. Roy.

Soc. London Ser.

A 302 (1968), 491–500.[Ba]V. Bargmann, Irreducible unitary representations of the Lorentz group, Ann.

of Math. (2) 48 (1947), 568–640.[BB]A.

Beilinson and J. Bernstein, Localisation de g-modules, C. R. Acad. Sci.

Paris S´er. IMath.

292 (1981), 15–18.[BK]J-L. Brylinski, and M. Kashiwara, Demonstration de la conjecture de Kazhdan-Lusztigsur les modules de Verma, C. R. Acad.

Sci. Paris S´er.

I Math. 291 (1980), 373–376.[Br]T.

Branson, Group representations arising from Lorentz conformal geometry, J. Funct.Anal. 74 (1987), 199–241.[BS]M.

Baldoni-Silva, The unitary dual of Sp(n, 1), n ≥2, Duke Math. J.

48 (1981), 549–583.[CC]L. Casian and D. Collingwood, Weight filtrations for induced representations of realreductive groups, Adv.

Math. 73 (1989), 79–144.[Cl]D.

Collingwood, Representations of rank one Lie groups, Pitman Res. Notes Math.

Ser.,vol. 137, Pitman Publishing, Boston, MA, 1985.[Cw]M.

Cowling, Unitary and uniformly bounded representations of some simple Lie groups,Harmonic Analysis and Group Representations, C.I.M.E. II ciclo 1980, Liguori editore,Naples, 1982, pp.

49–128.[DGN]Y. Dothan, M. Gell-mann, and Y. Ne’eman, Series of hadron energy levels as represen-tations of non-compact groups, Phys.

Lett. 17 (1965), 148–151.[Di]J.

Dixmier, Repr´esentations int´egrables du groupe de De Sitter, Bull. Soc.

Math. France89 (1961), 9–41.[EHW]T.

Enright, R. Howe, and N. Wallach, A classification of unitary highest weight modules,Representation Theory of Reductive Lie Groups (P. Trombi, ed. ), Birkhauser, Boston,MA, 1983, pp.

97–143.[En]M. Englefield, Group theory and the Coulomb problem, Wiley-Interscience, New York,1972.

[EPWW] T. Enright, R. Parthasarathy, N. Wallach, and J. Wolf, Unitary derived functor moduleswith small spectrum, Acta Math. 154 (1985), 105–136.[Fa]J.

Faraut, Distributions spheriques sur les espaces hyperboliques, J. Math.

Pures Appl.58 (1979), 369–444.[FJ]M. Flensted-Jensen, Analysis on non-Riemannian Symmetric Spaces, CBMS Reg.

Conf.Ser. in Math., vol.

61, Amer. Math.

Soc., Providence, RI, 1986, pp. 1–77.[FR]J.

Fischer and R. Raczka, Degenerate representations of non-compact unitary groups.II, Comm. Math.

Phys. 4 (1967), 8–21.[Fr]C.

Fronsdal, Infinite multiplets and the hydrogen atom, Phys. Rev.

(3) 156 (1967),1665–1677.[GN]I. M. Gel’fand and M. A. Na˘imark, Unit¨are darstellungen der klassischen gruppen,Acadamie-Verlag, Berlin, 1957.[Gr]K.

Gross, The dual of a parabolic subgroup and a degenerate principal series of Sp(n, C),Amer. J.

Math. 93 (1971), 398–428.

72R. E. HOWE AND E. C. TAN[Gu]A. Guillemonat, On some semi-spherical representations of a hermitian symmetric pairof tubular type, Math.

Ann. 246 (1980), 93–116.

[HC1]Harish-Chandra, Infinite irreducible representations of the Lorentz group, Proc. Roy.Soc.

London Ser. A 189 (1947), 372–401.

[HC2], Representations of a semisimple Lie group on a Banach space. I, Trans.

Amer.Math. Soc.

75 (1953), 185–243.[Hi1]T. Hirai, On the irreducible representations of the Lorentz group of n-th order, Proc.Japan Acad.

38 (1962), 258–262. [Hi2], Classification and the characters of irreducible representations of SU(p, 1),Proc.

Japan Acad. 42 (1966), 907–921.[Ho1]R.

Howe, On some results of Strichartz and of Rallis and Schiffman, J. Funct. Anal.

32(1979), 297–303. [Ho2], On a notion of rank for unitary representations of the classical groups, Har-monic Analysis and Group Representations, C.I.M.E.

II ciclo 1980, Liguori editore,Naples, 1982, pp. 223–332.

[Ho3], Remarks on classical invariant theory, Trans. Amer.

Math. Soc.

313 (1989),539–570. [Ho4], A century of Lie Theory, Mathematics into the Twenty-first Century (F. Brow-der, ed.

), American Mathematical Society Centennial Publications, vol. 2, Amer.

Math.Soc., Providence, RI, 1992, pp. 201–421.[Ja]N.

Jacobson, Lectures in Abstract algebra, Volume II, D. Van Nostrand, Princeton, NJ,1953.[Jo]K. D. Johnson, Composition series and intertwining operators for the spherical principalseries.

II, Trans. Amer.

Math. Soc.

215 (1976), 269–283.[JV]H. P. Jakobsen and M. Vergne, Wave and Dirac Operators and representations of theconformal group, J. Func.

Anal. 24 (1977), 52–106.[JW]K.

Johnson and N. Wallach, Composition series and intertwining operators for thespherical principal series, Trans. Amer.

Math. Soc.

229 (1977), 137–173.[KL]D. Kazhdan and G. Lusztig, Representations of Coxeter groups and Hecke algebras,Invent.

Math. 53 (1979), 165–184.[KG]A.

U. Klimyk and A. M. Gavrilik, The representations of the groups U(n, 1) andSO0(n, 1), preprint ITP-76-39 E, Institute for Theoretical Physics Kiev, USSR, 1976.[Kn]A. W. Knapp, Representation theory of semisimple groups, an overview based on exam-ples, Princeton Univ.

Press, Princeton, NJ, 1986.[Ko]B. Kostant, On the existence and irreducibility of certain series of representations, Bull.Amer.

Math. Soc.

75 (1969), 627–642.[KR]S. Kudla and S. Rallis, Degenerate principal series and invariant distributions, Israel J.Math.

69 (1990), 25–45.[KV1]M. Kashiwara and M. Vergne, On the Segal-Shale-Weil representations and harmonicpolynomials, Invent.

Math. 44 (1978), 1–47.

[KV2],Functions on the Shilov boundaryof the generalised half plane,Non-commutative Harmonic Analysis, Lecture Notes in Math., vol. 728, Springer, 1979, pp.136–176.[Ma]G.

W. Mackey, Induced representations of locally compact groups II, Ann. of Math.

(2)58 (1953), 193–221.[Mo]V. F. Molcanov, Analogue of the Plancherel Formula for hyperboloids, Soviet Math.Doklady 9 (1968), 1387–1385.[Na]M.

A. Na˘ımark, Linear representations of the Lorentz group, Amer. Math.

Soc. Transl.Ser.

2, vol. 6, Amer.

Math. Soc., Providence, RI, 1952, pp.

379–458.[Ni]K. Nishiyama, Algebraic structures on virtual characters of a semisimple Lie group,Representations of Lie groups, Kyoto, Hiroshima, 1986, Adv.

Stud. Pure Math., vol.

14,1988, pp. 417–468.[On]E.

Onofri, Dynamical quantization of the Kepler manifold, J. Math.

Phys. 17 (1976),401–408.[Or]B.

Orsted, Conformally invariant differential equations and projective geometry, J.Funct. Anal.

44 (1981), 1–23.[PS1]S. Paneitz and I. Segal, Analysis in space-time bundles, I: General considerations and

DEGENERATE PRINCIPAL SERIES73the scalar bundle, J. Funct. Anal.

47 (1982), 78–142. [PS2], Analysis in space-time bundles, II: The spinor and form bundles, J. Funct.Anal.

49 (1982), 335–414.[P]S. Paneitz, Analysis in space-time bundles, III: Higher spin bundles, J. Funct.

Anal. 54(1983), 18–112.[Re]J.

Repka, Tensor products of holomorphic discrete series and representations, Canad.J. Math., 31 (1979), 863–844.[Ro]W.

Rossman, Analysis on real hyperbolic spaces, J. Funct. Anal.

30 (1978), 448–477.[RS]S. Rallis and G. Schiffmann, Discrete spectrum of the Weil representation, Bull.

Amer.Math. Soc.

83 (1977), 267–270.[Sa]S. Sahi, The Capelli Identity and unitary representations, Compositio Math.

(to appear).[Sp1]B. Speh, Degenerate series representations of the universal covering group of SU(2, 2),J. Funct.

Anal. 33 (1979), 95–118.

[Sp2], The unitary dual of GL(3, R) and GL(4, R), Math. Ann.

258 (1981), 113–133.[SS]A. Salam and J. Strathdee, Relativistic U(6, 6) Theory, Phys.

Rev. 148 (1966), 1352–1358.[Sc]H.

Schlichtkrull, Eigenspaces of the Laplacian on hyperbolic spaces: composition seriesand integral transforms, J. Funct. Anal.

70 (1987), 194–219.[Se]J. Sekiguchi, Eigenspaces of the Laplace-Beltrami operator on a hyperboloid, NogoyaMath.

J. 79 (1980), 151–185.[Sta]R.

Stanley, Enumerative Combinatorics, vol. 1, Wadsworth and Cole, Monterey, CA,1986.[St1]R.

S. Strichartz, Harmonic analysis on hyperboloids, J. Funct. Anal.

12 (1973), 341–383. [St2], Harmonic analysis as spectral theory of laplacians, J. Funct.

Anal. 87 (1989),51–148.[SV]B.

Speh and D. A. Vogan, Reducibility of generalised principal series representations,Acta Math. 145 (1980), 227–299.[SW]S.

Sternberg and J. Wolf, Hermitian Lie algebras and metaplectic representations. I,Trans.

Amer. Math.

Soc. 238 (1978), 1–43.[Te]A.

Tengstrand, Distributions invariant under an orthogonal group of arbitrary signature,Math. Scand.

8 (1960), 201–218.[Tk]R. Takahashi, Sur les representations unitaire des groupes de Lorentz generalises, Bul.Soc.

Math. France 91 (1963), 289–435.[Vi]N.

J. Vilenkin, Special functions and the theory of group representations, Izdat NaukaMoscow 1965, Transl. Math.

Monographs, vol. 22, Amer.

Math. Soc., Providence, RI,1968.[Vo1]D.

Vogan, The Gelfand-Kirillov dimension for Harish-Chandra modules, Invent. Math.48 (1978), 75–98.

[Vo2], Representations of real reductive groups, Progress in Math., vol. 15, Birkhauser,Boston, MA, 1981.

[Vo3], Irreducible characters of semisimple Lie groups. III, Proof of the Kazhdan-Lusztig conjectures in the integral case, Invent.

Math. 71 (1983), 381–417.

[Vo4], Unitary representations of reductive Lie groups, Ann. Math.

Stud., vol. 118,Princeton Univ.

Press, Princeton, NJ, 1987.[Wa]N. Wallach, Real Reductive Groups.

I, Pure Appl. Math., vol.

132, Academic Press, SanDiego, CA, 1988.[Zh]D. P. ˇZhelobenko, Compact Lie groups and their representations, Transl.

Math. Mono-graphs, vol.

40, Amer. Math.

Soc., Providence, RI, 1973.Department of Mathematics, Yale University, New Haven, Connecticut 06520E-mail address: howe@lom1.math.yale.eduDepartment of Mathematics, National University of Singapore, Kent Ridge, Singa-pore 0511, SingaporeE-mail address: mattanec@nusvm.bitnet

---

출처: arXiv:9301.214 • 원문 보기