ANTON DEITMAR
Received 18 May 2005; Revised 26 June 2006; Accepted 5 July 2006
It is shown that the space of invariant trilinear forms on smooth representations of a semisimple Lie group is finite dimensional if the group is a product of hyperbolic groups. Explicit upper bounds are given which are attained in the case of induced representations. Applications to automorphic coefficients are given.
Copyright © 2006 Hindawi Publishing Corporation. All rights reserved.
1. Introduction
LetG=PGL2(R) and letπ1,π2,π3be irreducible admissible smooth representations ofG.
Then the space ofG-invariant trilinear forms onπ1×π2×π3is at most one dimensional.
This has, in different contexts, been proved by Loke [9], Molˇcanov [10], and Oksak [11]. In this paper, we ask for such a uniqueness result in the context of arbitrary semisimple groups. We give evidence that for a given groupG, uniqueness can only hold ifGis locally a product of hyperbolic groups. For such groups, we show uniqueness and for spherical vectors, we compute the invariant triple products explicitly.
By a conjecture of Jacquet’s, which has been proved in [3], triple products on GL2
are related to special values of automorphicL-functions, see also [2,4,6,7]. The con-jecture/theorem says that the existence of nonzero triple products is equivalent to the nonvanishing of the corresponding tripleL-function at the center of its functional equa-tion.
The uniqueness of triple products in the PGL2-case mentioned above has been used
in [1] to derive new bounds for automorphicL2-coefficients. This can also be done for
higher-dimensional hyperbolic groups, but, with the exception of the case treated in [1], the results do not exceed those in [8]. For completeness, we include these computations in the appendix.
2. Representations and integral formulae
LetG be a connected semisimple Lie group with finite center. Fix a maximal compact subgroup K. LetGandK denote their unitary duals, that is, the sets of isomorphism
Hindawi Publishing Corporation
International Journal of Mathematics and Mathematical Sciences Volume 2006, Article ID 48274, Pages1–22
classes of irreducible unitary representations ofG, respectively,K. Letπbe a continuous representation ofGon a locally convex topological vector spaceVπ. LetVπbe the space of all continuous linear forms onVπand letVπ∞be the space of smooth vectors, that is,
Vπ∞=v∈Vπ:x−→α
π(x)vis smooth∀α∈Vπ. (2.1)
The representationπis called smooth ifVπ=Vπ∞.
A representation (π,Vπ) is called admissible if for eachτ∈K, the space HomK(Vτ,Vπ) is finite-dimensional. LetGadmbe the admissible dual, that is, the set of infinitesimal
iso-morphism classes of irreducible admissible representations. A representationπ inGadm
is called a class one or spherical representation if it containsK-invariant vectors. In that case, the spaceVK
π ofK-invariant vectors is one-dimensional. This is trivial for principal series representations (see below) and follows generally from Casselman’s subrepresen-tation theorem which says that everyπ∈Gadmis equivalent to a subrepresentation of a
principal series representation.
The Iwasawa decompositionG=ANK gives smooth maps
a:G−→A,
n:G−→N,
k:G−→K,
(2.2)
such that for everyx∈G, one hasx=a(x)n(x)k(x). As an abbreviation, we also define
an(x)=a(x)n(x). LetgR,aR,nR,kRdenote the Lie algebras ofG,A,N,Kand letg,a,n, kbe their complexifications.
Forx∈Gandk∈K, we define
kxdef=k(kx). (2.3)
Lemma 2.1. The rulek→kxdefines a smooth (right) group action ofGonK.
Proof. The mapkgives a diffeomorphismAN\G→Kand the action under consideration is just the natural right action ofGonAN\G.
Lemma 2.2. For f ∈C(K) andy∈G, one has the integral formula
K f(k)dk=
Ka(k y)
2ρfkydk, (2.4)
or
K f
kydk=
Ka
k y−12ρf(k)dk. (2.5)
Hereρ∈a∗is the modular shift, that is,a2ρ=det(a|n). Proof. The Iwasawa integral formula implies that
Gg(x)dx=
AN
Let f ∈C(K) and choose a functionη∈Cc(AN) such thatη≥0 andANη(an)da dn=1. Letg(x)=η(an(x))f(k(x)). Theng∈Cc(G) and
Gg(x)dx=
ANη(an)dan
K f(k)dk=
K f(k)dk. (2.7) On the other hand, sinceGis unimodular, this also equals
Gg(xy)dx=
Gη
an(xy)fk(xy)dx
=
AN
Kη
an(ank y)fk(k y)dank
=
K ANη
an(ank y)dan
fkydk
=
K ANη
anan(k y)dan
fkydk
=
Ka(k y)
2ρ
ANη(an)dan
fkydk
=
Ka(k y)
2ρfkydk.
(2.8)
The second assertion follows from the first by replacing f with f(k)= f(ky) and theny
withy−1.
LetMbe the centralizer ofAintersected withK, thenP=MANis a minimal parabolic subgroup ofG. The inclusion mapKGinduces a diffeomorphismM\K→P\Gand in this way, we get a smoothG-action onM\K. An inspection shows that this action is given byMk→Mkxfork∈K,x∈G.
3. Trilinear products
Let π1,π2,π3 be three admissible smooth representations of the group G and let᐀: Vπ1×Vπ2×Vπ3→Cbe a continuousG-invariant trilinear form, that is,
᐀π1(x)v1,π2(x)v2,π3(x)v3
=᐀v1,v2,v3
(3.1)
for allvj∈Vπjand everyx∈G.
We want to understand the space of all trilinear forms᐀as above. In this paper, we will only consider principal series representations, the general case will be considered later. So we assume thatπ1,π2,π3are principal series representations. This means that there are
given a minimal parabolicP=MAN, irredicible representationsσj∈M, andλj∈a∗for
j=1, 2, 3. Each pair (σj,λj) induces a continuous group homomorphismP→GL(Vσj) by
man→aλj+ρσ(m), which in turn defines aG-homogeneous vector bundleE
σj,λjoverP\G.
The representationπjis theG-representation on the space of smooth sectionsΓ∞(Eσj,λj)
of that bundle. In other words,πjlives on the space of allC∞functions f :G→Vσjwith
f(manx)=aλj+ρσ
for allm∈M,a∈A,n∈N,x∈G. The representationπj is defined byπj(y)f(x)=
f(xy). Every such f is uniquely determined by its restriction toKwhich satisfiesf(mk)=
σj(m)f(k), that is, f is a section of theK-homogeneous bundleEσj onM\Kinduced by
σj. So the representation space can be identified withVπj∼=Γ∞(Eσj). Thus a trilinear form
᐀is a distribution on the vector bundleEσ=Eσ1Eσ2Eσ3overM\K×M\K×M\K. Heredenotes the outer tensor product.
For f1,f2,f3∈C∞(M\K), we write᐀(f1,f2,f3) for the expression
M\K×M\K×M\Kφ
k1,k2,k3
f1
k1
f2
k2
f3
k3
dk1dk2dk3, (3.3)
whereφis the kernel of᐀.
The groupGis called a real hyperbolic group if it is locally isomorphic to SO(d, 1) for somed≥2.
OnY=(P\G)3, we consider theG3-homogeneous vector bundleE
σ,λgiven byEσ,λ=
Eσ1,λ1Eσ2,λ2Eσ3,λ3. NextY can be viewed as aG-space via the diagonal action and so
Eσ,λbecomes aG-homogeneous line bundle onY.
We are going to impose the following condition on the induction parametersλ1,λ2, λ3. We assume that3j=1εj(λj+ρ)=0 for any choice ofεj∈ {±1}. In other words, this means that
(i)λ1+λ2+λ3+ 3ρ=0,
(ii)λ1+λ2−λ3+ρ=0,
(iii)λ1−λ2−λ3−ρ=0.
Theorem 3.1. Assume the parametersλ1,λ2,λ3satisfy the above condition. LetY be the G-space (P\G)3. If there is an openG-orbit inY, then the dimension of the space of invariant
trilinear forms on smooth principal series representations is less than or equal to
o
dimσ1⊗σ2⊗σ3
Mo
, (3.4)
where the sum runs over all open orbitsoandMois the stabilizer group of a point in the orbit
owhich is chosen so thatMois a subgroup ofM. If all induction parameters are imaginary (unitary induction), then there is equality.
In particular, ifπ1,π2,π3are class-one representations, then the dimension is less than or
equal to the number of open orbits inY.
There is an open orbit if and only ifG is locally isomorphic to a product of hyperbolic groups.
ForG=SO(2, 1)0, the number of open orbits is 2, forG=SO(2, 1) orG=SO(d, 1)0, d≥3, the number is 1. Here SO(d, 1)0is the connected component of the Lie group SO(d, 1).
The Proof is based on the following lemma.
Lemma 3.2. LetGbe a Lie group andH a closed subgroup. LetX=G/Hand letE→Xbe a smoothG-homogeneous vector bundle. Let᐀be a distribution onE, that is, a continuous linear form on Γ∞c (E). Suppose that᐀ isG-invariant, that is,᐀(g·s)=᐀(s) for every
s∈Γ∞
Let (σ,Vσ) be the representation ofHon the fibreEeHand let (σ∗,Vσ∗) be its dual. Then the space of allG-invariant distributions onEhas dimension equal to the dimension dimVσH∗ ofH-invariants. So in particular, ifσis irreducible, this dimension is zero unlessσis trivial, in which case the dimension is one.
Proof. Equation᐀(g·s)=᐀(s), that is,g·᐀=᐀for allg∈Gimplies thatX·᐀=0 for everyX∈gR, the real Lie algebra ofG. LethRbe the Lie algebra ofHand choose a
complementary spacepRforhRsuch thatgR=hR⊕pR. LetX1,. . .,Xnbe a basis ofVand let
D=X2
1+X22+···+Xn2∈U g
R. (3.5)
We show thatDinduces an elliptic differential operator onE. ByG-homogeneity, it suf-fices to show this at a single point. So letP=exp(pR). In a neighborhoodU of the unit
inG, there are smooth mapsh:U→Handp:U→Psuch thath(x)p(x)=xforx∈U. The sections ofEcan be identified with the smooth mapss:G→Vσ withs(hx)=
σ(h)s(x) forh∈Handx∈G. We can attach to each sectionsa map fsonpwith values inVσ byfs(Y)=s(exp(Y)). The action ofX∈pRon the sectionsis described by
fXs(Y)= d
dt
t=0
sexp(Y) exp(tX)
= d
dt
t=0
σhexp(Y) exp(tX)spexp(Y) exp(tX).
(3.6)
LetAX(Y)=d/dt|t=0σ(h(exp(Y) exp(tX))∈End(Vσ). Then the Leibniz rule implies that
fXs(Y)=AX(Y)fs(Y) +X fs(Y). (3.7)
The first summand is of order zero and the second is of order one. Moreover, the sec-ond summand atY=0 coincides with the coordinate-derivative in the direction ofX. This implies that the leading symbol ofDateHisξ12+···+ξn2and soDis elliptic. The distributional equationD᐀=0 then implies that᐀is given by a smooth section.
For the second assertion of the lemma, recall that aG-invariant section is uniquely determined by its restriction to the pointeHwhich must be invariant underH.
For the proof of the theorem, we will need to investigate the G-orbit structure of
Y =(P\G)3. First note that since the mapMk→Pk is aK-isomorphism fromM\K
toP\G, theK-orbit of every y∈Y contains an element of the form (y1,y2, 1). Hence
the P=MAN-orbit structure of (P\G)2 is the same as the G-orbit structure ofY. By
the Bruhat decomposition, the P-orbits in P\G are parametrized by the Weyl group
W=W(G,A), where the unique open orbit is given byPw0P, herew0 is the long
ele-ment of the Weyl group. Note that theP-stabilizer ofPw0∈P\GequalsAM. This
P\Gof maximal dimension via the mapPxAM→(x,w0, 1)·G. Again by Bruhat
decom-position, it follows that the latter are contained in the open cellPw0P=Pw0N. So the G-orbits of maximal dimension inY are in bijection to theAM-orbits inNof maximal dimension, whereAMacts via the adjoint action. The exponential map exp :nR→N is
anAM-equivariant bijection, so we are finally looking for theAM-orbit structure of the linear adjoint action onnR.
We will now prove that there is an open orbit if and only ifGis locally a product of real hyperbolic groups. So suppose thatYcontains an open orbit. ThennRcontains an open AM-orbit, sayAM·X0. Letφ+be the set of all positive restricted roots on a=Lie(A).
DecomposenRinto the root spaces
nR=
α∈φ+
nR,α. (3.8)
On eachnR,α, install anM-invariant norm · α. This is possible sinceM is compact.
Consider the map
ψ:nR−→
α∈φ+ R,
x−→
α xα.
(3.9)
Since the orbit AM·X0 is open, the image ψ(AM·X0) of the orbit must contain a
nonempty open set. Away from the set{X∈nR:∃α:xα=0}, the mapψcan be chosen differentiable. Since the norms are invariant underM, one gets a smooth map
A−→R|φ+|
a−→ψa·X0
, (3.10)
whose image contains an open set. This can only happen if the dimension ofAis at least as big as|φ+|and the latter implies thatGis locally a product of real rank-one groups.
Now by Araki’s table (see [5, pages 532–534]), one knows that these real rank-one groups must all be hyperbolic, because otherwise there would be two different root lengths.
For the converse direction, letGbe locally isomorphic to SO(d, 1). We have to show that there is an openAM-orbit innR. This, however, is clear as the action ofAMonnRis
the natural action ofR×+ ×SO(d−1) onRd−1, hence there are two orbits, the zero orbit
and one open orbit.
We will now show that if there is an open orbit, then there are no invariant distri-butions supported on lower-dimensional orbits. For this, it suffices to consider the case
G=SO0(d, 1). For simplicity, we only consider the trivial bundle, that is, functions in-stead of sections. The general case is similar. The orbit structure is as follows. Ford=2, the groupM=SO(d−1) is trivial, and so there are two openAM-orbits inN∼=Rin this case. Ifd >2, there is only one open orbit. Since the proof is very similar in the case
d=2, we will now restrict ourselves to the cased >2. The open orbit [w0n0,w0, 1], given
contain in their closures the orbit [1, 1, 1],
w0n0,w0, 1
∪ ∪
w0,w0, 1
w0, 1, 1
∪ ∪
[1, 1, 1]
(3.11)
Letᏻ=[w0n0,w0, 1] be the open orbit. Letx1=(w0,w0, 1), then one has [w0,w0, 1]= x1·G. ConsiderA∼=R×+as a subset ofRsuitably normalized, then one can write
x1=lim
a→0x0·am. (3.12)
Let᐀be aG-invariant distribution supported on the closure of the orbit ofx1. Since᐀
isG-invariant, it satisfiesX·᐀=0 for everyX∈g. Hence the wave front setWF(᐀)⊂
T∗Y is aG-invariant subset of the normal bundle of the manifoldx1·G. This implies
that᐀is of order zero along the manifoldx1·G. By theG-invariance, it follows that᐀is
of the form
᐀(f)=
x1·G
Dxfx1
dx+R, (3.13)
whereRis supported in [1, 1, 1]. Further,Dis a differential which we can assume to be
G-equivariant. ThenD(f)(x1) is of the form
D1(m)fx0·am|a=0, (3.14)
whereD1(m) is a differential operator in the variablea. SinceDisG-equivariant, we may
replace f witha0f for somea
0∈A. Sincex1a0=x1, we get that the above is the same as
D1(m)fx0·aa0m|a=0. (3.15)
This implies thatD1must be of order zero and so᐀is of order zero. Restricted to the orbit x1·G∼=AM\G, the distribution᐀is given by an integral of the form
AM\Gφ(y)f(x1y)d y.
(Note that we use the notation without the dot again.) Invariance implies thatφis con-stant. If᐀is nonzero, then y→ f(x1y) must be left invariant underAM, which implies
thatλ1+λ2−λ3+ρ=0, a case we have excluded. So ᐀must be zero. This shows that
any invariant distribution which is zero on the open orbit also vanishes onx1·G. The
remaining orbits are dealt with in a similar fashion. To prove the theorem, it remains to show the existence of invariant distributions in the case of unitary parameters. For this, we change our point of view and consider sections ofLλno longer as functions onY, but as functions onG3with values inV
σ=Vσ1⊗Vσ2⊗Vσ3 which spit outaλ+ρσ(m) on the left. We induce this in the notation by writing f(x0y) instead of f(x0·y). On a given
is unique up to scalars and is given by
᐀stf =α
Gf
x0yd y
, (3.16)
whereαis a linear functional on the space ofMo-invariants. In order to show that this extends to a distribution onY, we need to show that the defining integral converges for all f ∈Γ∞(E
σ,λ). This integral equals
Gf
x0y
d y=
Gf
w0n0y,w0y,y
d y
=
Ga
w0n0y
ρ+λ1
aw0y
ρ+λ2
a(y)ρ+λ3fkx
0y
d y.
(3.17)
Since f is bounded onK3, it suffices to show the following lemma.
Lemma 3.3. LetG=SO(d, 1)0and letk
0=k(w0n0). Then
Ga
w0n0y
ρ
aw0y
ρ
a(y)ρd y <∞. (3.18)
Conjecture 3.4. The assertion of the lemma should hold for any semisimple groupGwith finite center andn0∈Ngeneric.
The conjecture would imply that ifGis not locally a product of hyperbolic groups, then the space of invariant trilinear forms on principal series representations is infinite-dimensional.
Proof ofLemma 3.3. Replace the integral overGby an integral overANK using the Iwa-sawa decomposition. Sincea(xk)=a(x) forx∈Gandk∈K, theK-factor is irrelevant and we have to show that
ANa
k0anρaw0anρaρda dn <∞. (3.19)
Now write w0n0 =ank0. Then k0an=(an)−1w0n0an, and therefore a(k0an)=
(a)−1a(w
0n0an), so it suffices to show that
ANa
w0n0an
ρ
aw0an
ρ
aρda dn <∞. (3.20)
Next note thatw0a=a−1w0and so we havea(w0an)ρ=a−ρa(w0n)ρas well asa(w0n0an)ρ= a−ρa(w0na0n)ρ, wherena0=a−1n0a. We need to show that
ANa
w0na0n
ρ
aw0n
ρ
a−ρda dn <∞. (3.21)
real matricesgwithgtJg=J. Writingg=A b c d
withA∈Matd(R), this amounts to
AtA−ctc=1,
Atb−ctd=0,
d2−btb=1.
(3.22)
The connected component SO(d, 1)0 consists of all matricesg as above withd >0. The
maximal compact subgroupKcan be chosen to be
SO(d) 1
(3.23)
andMas
⎛ ⎜ ⎝
SO(d−1) 0 1 0
0 0 1
⎞ ⎟
⎠. (3.24)
Further, we can chooseAandNas follows:
A= ⎧ ⎪ ⎨ ⎪ ⎩ ⎛ ⎜ ⎝ 1 α β β α ⎞ ⎟
⎠:α >0,α2−β2=1
⎫ ⎪ ⎬ ⎪ ⎭, N= ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩
n(x)= ⎛ ⎜ ⎜ ⎜ ⎜ ⎝
1 −x x
xt 1−|x|2
2 D
|x|2
2
xt −|x|2
2 1 +
|x|2
2 ⎞ ⎟ ⎟ ⎟ ⎟
⎠:x∈Rd−
1 ⎫ ⎪ ⎪ ⎪ ⎪ ⎪ ⎬ ⎪ ⎪ ⎪ ⎪ ⎪ ⎭ , (3.25)
where we have written|x|2=x2
1+x22+···+xd−2 1. Note that the Lie algebra ofAis
gen-erated byH=#10 1 1 0
$
. One derives an explicit formula for theANK-decomposition. In particular, ifg=A b
c d
and
a(g)= ⎛ ⎜ ⎝ 1 α β β α ⎞ ⎟
⎠, (3.26)
then
a(g)ρ=(α+β)(d−1)/2=
d+bd
1 +b2
1+···+b2d−1
(d−1)/2
The Weyl representative element can be chosen to be
w0=
⎛ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ 1 . .. 1 −1 −1 1 ⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠ . (3.28)
So that withn=n(x) forx∈Rd−1, we have
aw0n
ρ =a ⎛ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝
∗ ∗ x1
∗ ∗ ... ∗ ∗ xd−2
∗ ∗ −xd−1
∗ ∗ −|x|2 2 ∗ ∗ 1 +|x|
2 2 ⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠ ρ = 1
1 +x12+···+xd−2 1
(d−1)/2 .
(3.29)
We choosen0=n(1, 0,. . ., 0) and get witha=exp(tH) that
aw0na0n
ρ =
1 1 +e−t+x12+x2
2+···+x2d−1
(d−1)/2
. (3.30)
Withn=d−1 anda−ρ=e−(n/2)t, our assertion boils down to
Rn
Re
−(n/2)t1 +|x|2−n/2#
1 +e−tv1+x2
$−n/2
dt dx < ∞. (3.31)
Consider first the casen=1 and the integral overx <0:
R
0
−∞e
−(1/2)t1 +x2−1/2#1 +e−t+x2$−1/2 dx dt = R ∞ 0 e
−(1/2)t1 +x2−1/2#
1 +x−e−t2$−1/2dx dt
=
R
∞
−e−te
−(1/2)t#1 +x+e−t2$−1/2
1 + (x)2−1/2dx dt.
(3.32)
Thus it suffices to show the convergence of
R
∞
0 e
−(1/2)t1 +x2−1/2#1 +x+e−t2$−1/2
Settingy=e−t, we see that this integral equals ∞
0 y
−1/21 +x2−1/21 + (x+y)2−1/2d y dx
= ∞
0
∞
x (y−x)
−1/21 +x2−1/21 +y2−1/2d y dx.
(3.34)
Since the mapping
x−→ ∞
x (y−x)
−1/21 +x2−1/21 +y2−1/2d y (3.35)
is continuous, the integral over 0< x <1 converges. It remains to show the convergence of
∞
1
∞
x (y−x)
−1/21 +x2−1/2
1 +y2−1/2d y dx
= ∞
1 x
1/2(v−1)−1/21 +x2−1/2
1 +v2x2−1/2dv dx
≤3−1/2 ∞
1 x
1/2(v−1)−1/21 +x2−1
1 +v2−1/2dv dx <∞.
(3.36)
Here we have used the substitution y=vx and the fact that fora,b≥1, one has (1 +
a)(1 +b)≤3(1 +ab).
Now for the casen >1, using polar coordinates, we compute
Rn−1
Re
−tn/2#1 +x2+x
r2 $−n/2
(1+)e−t+x2+xr2 $−n/2
dt dx dxr
=C
∞
0
Re
−tn/2rn−21 +x2+r2−n/2
(1+)e−t+x2+r2$−n/2dt dx dxr.
(3.37)
As above, we can restrict to the casex >1. We get that this equals ∞
0 y
(n/2)−1rn−21 +x2+r2−n/21 + (x+y)2+r2−n/2d y dx dr
= ∞
0
∞
x (y−x)
(n/2)−1rn−21 +x2+r2−n/21 +y2+r2−n/2d y dx dr,
(3.38)
which equals ∞
0
∞
1 (v−1)
n/2−1xn/2rn−21 +x2+r2−n/2
1 +v2x2+r2−n/2d y dx dr. (3.39)
As above, it suffices to restrict the integration to the domainx >1. So one considers ∞
0
∞
1 (v−1)
Choose 0< ε <1/2 and write
1 +x2+r2−n/2=1 +x2+r2ε−n/21 +x2+r2−ε
≤1 +r2ε−n/21 +x2−ε.
(3.41)
So our integral is less than or equal to
∞
0 r
n−21 +r2ε−n/2
dr (3.42)
times ∞
1 (v−1)
(n/2)−1xn/21 +x2−ε1 +v2x2−n/2dv dx
≤C
∞
1 (v−1)
(n/2)−1xn/21 +x2−ε1 +v2−n/21 +x2−n/2dv dx <∞.
(3.43)
4. An explicit formula
Letd≥2 andG=SO(d, 1)0 ifd >2. Ford=2, letG be the double coverSO(2, 1)% o∼= PGL2(R). ThenK=SO(d),M=SO(d−1) ford >2. Ford=2, we haveK∼=O(2) and M∼=Z/2Zand in all cases, we haveM\K∼=Sd−1, the (d−1)-dimensional sphere. For each λ∈a∗, lete
λ be the class-one vector in the associated principal series representationπλ given byeλ(ank)=aλ+ρ. Letλ,μ,ν∈a∗be imaginary. Let᐀
stbe the invariant distribution
onL(λ,μ,ν)considered in the last section. We are interested in the growth of᐀st(eλ,eμ,eν)
as a function inλ. First note that the Killing form induces a norm| · |ona∗.
We write᐀st(λ,μ,ν) for᐀st(eλ,eμ,eν) and identifyingaRtoRviaλ→λ(H0), we
con-sider᐀stas a function on (iR)3.
In this section, we will prove the following theorem.
Theorem 4.1. Forλ,μ,νimaginary,᐀st(λ,μ,ν) equals a positive constant times
Γ2(λ+μ−ν)+n/4Γ2(λ−μ+ν)+n/4Γ2(−λ+μ+ν)+n/4Γ2(λ+μ+ν)+n/4 Γ(2λ+n)/2Γ(2μ+n)/2Γ(2ν+n)/2 ,
(4.1)
wheren=d−1, so in particular, for fixed imaginaryμandν. Then, as|λ|tends to infinity, whileλis imaginary, one has the asymptotic
᐀st(λ,μ,ν)=cexp −π
2|λ|
|λ|(d/2)−2 1 +O 1
|λ|
, (4.2)
Proof. The asymptotic formula follows from the explicit expression and the well-known asymptotical formula,
Γ(σ+it)=√
2πexp −π 2|t|
|t|σ−(1/2) 1 +O 1
|t|
(4.3)
as|t| → ∞, where the real partσis fixed.
Now for the proof of the first assertion. The permutation groupS3in three letters acts
on (iR)3by permuting the coordinates. We claim that᐀
stis invariant under that action.
So letσ∈S3and letf =f1⊗f2⊗f3, then
᐀st
σ(f)=
Gf
σ−1x0·y
d y=
Gf
σ−1x0
·yd y. (4.4)
Since the open orbit is unique, there isyσ∈Gwithσ−1(x0)=x0·yσ, and so
᐀stσ(f)=
Gf
x0·yσyd y=
Gf
x0·yd y=᐀st(f). (4.5)
So in particular᐀st(λ,μ,ν) is invariant under permutations of (λ,μ,ν).
We have
᐀st
eλ,eμ,eν=
Ga
w0n0y
λ+ρ
aw0y
μ+ρ
a(y)ν+ρd y. (4.6)
Replace the integral overGby an integral overANK using the Iwasawa decomposition. Sincea(xk)=a(x) forx∈Gandk∈K, theK-factor is irrelevant and we have to compute
ANa
w0n0an
λ+ρ
aw0an
μ+ρ
aν+ρda dn. (4.7)
Note thatw0a=a−1w0, and so we havea(w0an)ρ=a−ρa(w0n) as well as a(w0n0an)= a−ρa(w0na0n), wherena0=a−1n0a. So the integral equals
ANa
ν−λ−μ−ρaw
0na0n
λ+ρ
aw0n
μ+ρ
da dn. (4.8)
Using the explicit approach of the last section, we see that᐀st(λ,μ,ν) equals
0
R
Rd−1e
t(ν−λ−μ−(d−1)/2)#1 +e−tv
1+x
2$−λ−(d−1)/2
1 +|x|2−μ−(d−1)/2dx dt, (4.9)
wherev1=(1, 0,. . ., 0).
Setn=d−1=1, 2, 3,. . .and use polar coordinates to compute forn >1 that
R
Rne
t(ν−λ−μ−(n/2))#1 +e−tv
1+x
2$−λ−(n/2)
equals (n−1) vol(Bn−1) times
R
∞
0 r
n−2et(ν−λ−μ−(n/2))#1 +e−t+x2
+r2$−λ−(n/2)1 +x2+r2−μ−(n/2)dr dx dt.
(4.11)
DefineIn(a,b,c) to be
R
∞
0 r
n−2et(c−a−b)#1 +e−t+x2
+r2$−a1 +x2+r2−bdr dx dt, (4.12)
andI1(a,b,c) to be equal to
Re
t(c−a−b)#1 +e−t+x2$−a
1 +x2−bdr dx dt. (4.13)
Then, for alln∈N,
᐀st(λ,μ,ν)=cnIn λ+n
2,μ+
n
2,ν+
n
2
, (4.14)
wherec1=1 andcn=(n−1) vol(Bn−1) forn >1. So in particular,Inis invariant under permutations of the arguments.
Note that
∂ ∂r
&#
1 +e−t+x2+r2$−a1 +x2+r2−b' (4.15)
equals
−2ar#1 +e−t+x2+r2$−a−11 +x2+r2−b−2br#1 +e−t+x2
+r2$−a1 +x2+r2−b−1,
(4.16)
andrn−2=∂/∂r[rn−1/n−1]. So, integrating by parts, forn >1 we compute
In(a,b,c)= 2a
n−1In+2(a+ 1,b,c+ 1) + 2b
n−1In+2(a,b+ 1,c+ 1). (4.17)
To get a similar result forI1, note that
#
1 +e−t+x2$−a1 +x2−b (4.18)
equals
−#1 +e−t+x2+r2$−a1 +x2+r2−b|r=∞ r=0
= − ∞
0 ∂ ∂r
&#
This implies that
I1(a,b,c)=2aI3(a+ 1,b,c+ 1) + 2bI3(a,b+ 1,c+ 1). (4.20)
LetIn(a,b,c) be the same asIn(a,b,c) except that there is a factorxin the integrand, that is,
R
∞
0 r
n−2et(c−a−b)#1 +e−t+x2
+r2$−a1 +x2+r2−bx dr dx dt. (4.21)
Replacingxwith−xfirst and then withx+e−tyields
In(a,b,c)= −In(b,a,c)−In(b,a,c−1). (4.22)
The last equation also holds forn=1. Integration by parts gives
Re
t(c−a−b)#1 +e−t+x2
+r2$−adt
= 2a
a+b−c
Re
t(c−a−b)#1 +e−t+x2
+r2$−a−1e−2t+xe−tvdt.
(4.23)
So,
In(a,b,c)= 2a
a+b−c
In(a+ 1,b,c−1) +In(a+ 1,b,c)
. (4.24)
Using (4.22) yields
In(a,b,c)= 2a
c−a−bI
n(b,a+ 1,c), (4.25) or
In(x,y,z)=
z−x−y+ 1
2y−2 In(y−1,x,z). (4.26)
Plugging this into (4.22), one gets
c−a−b+ 1
2b−2 In(b−1,a,c)=
a+b−c−1
2a−2 In(a−1,b,c)−In(b,a,c−1). (4.27) Replacing (a,b,c) with (a+ 1,b+ 1,c+ 1) gives
c−a−b
2b In(b,a+ 1,c+ 1)=
a+b−c
2a In(a,b+ 1,c+ 1)−In(b+ 1,a+ 1,c), (4.28)
or, using the invariance ofIn,
c−b−a
2ab
aIn(a+ 1,b,c+ 1) +bIn(a,b+ 1,c+ 1)= −In(a+ 1,b+ 1,c), (4.29)
and hence
aIn+2(a+ 1,b,c+ 1) +bIn+2(a,b+ 1,c+ 1)= 2ab
Using (4.17) and (4.20), one gets forn >1 that
In(a,b,c)= 4ab
a+b−c
1
n−1In+2(a+ 1,b+ 1,c),
I1(a,b,c)= 4ab
a+b−cI3(a+ 1,b+ 1,c).
(4.31)
Finally we arrive at
In+2(a+ 1,b+ 1,c+ 1)=a+b−c−1
4ab (n−1)In(a,b,c+ 1) (4.32)
forn >1 and
I3(a+ 1,b+ 1,c+ 1)=a+b−c−1
4ab I1(a,b,c+ 1). (4.33)
These formulae amount to
᐀n+2
st (λ,μ,ν)=dnλ+μ−ν+ (n/2)−1 (2λ+n)(2μ+n) ᐀
n
st(λ,μ,ν+ 1), (4.34)
wheredn=(n−1)cn+2/cnifn >1 andd1=c3. A calculation using the functional
equa-tion of the Gamma funcequa-tion,Γ(z+ 1)=zΓ(z), shows that the right-hand side of the claim in the theorem satisfies the same equation asnis replaced byn+ 2. So the claim of the theorem fornimplies the same claim forn+ 2. Note that [1, formula (A.5)] implies the theorem forn=1, where one has to take into account that in [1], a different normaliza-tion is used. To get our formula from theirs, one has to replaceλjby−2λjin [1]. To finish the proof of the theorem, it therefore remains to show the claim forn=2.
To achieve this, we proceed in a fashion similar to [1]. First note that the groupG= SL2(C) is a double cover of SO(3, 1)0, so we might as well use this group for the
computa-tion. Forλ∈C, letVλdenote the space of all smooth functions f onC2withf(az,aw)= |a|−2(λ+1)f(z,w) for everya∈C×. ThenG=SL2(C) acts on the spaceVλviaπλ(g)f(z,w)=
f((z,w)g). This is the principal series representation with parameterλ. Forλ=1, there is aG-invariant continuous linear functionalL:V1→C, which is unique up to scalars and
is given by
L(f)=
S3f(x)dx, (4.35)
where the integral runs over the standard sphereS3⊂C2 with the volume element
in-duced by the standard metric on C2∼=R4. Since S3 equals (0, 1)K, where K=SU(2)
is the maximal compact subgroup of G, the theory of induced representations shows that this functional is indeed invariant underG and is unique with that property. Ifλ
is imaginary, then the representationπλis preunitary, the inner product being given by f,g =L(f g). To compute᐀2
st(λ1,λ2,λ3), we will describe this functional on the space Vλ1⊗Vλ2⊗Vλ3. Letω:C
2→Cbe given byω(a,b)=a
1b2−a2b1the bilinearG-invariant
onC2. LetK
λ1,λ2,λ3(v1,v2,v3) be the invariant kernel ω
v1,v2
(−λ1+λ2+λ3−1)
ωv1,v3
(λ1−λ2+λ3−1)
ωv2,v3
(λ1+λ2−λ3−1)
This kernel is invariant under the diagonal action ofGand is homogeneous of degree 2(λj−1) with respect to the variablevj. From this point, the computations run like in [1, formulae (A.4) and (A.5)]. The theorem follows.
Appendix
Automorphic coefficients
Let Γ⊂G be a uniform lattice and consider the right regular representation of G on
L2(Γ\G), which decomposes as a direct sum,
L2(Γ\G)=
π∈G
L2(Γ\G)(π), (A.1)
where the isotypic componentL2(Γ\G)(π) is zero forπoutside a countable set ofπ∈G
and is always isomorphic to a sum of finitely many copies ofπ. By the Sobolev lemma, one hasL2(Γ\G)∞⊂C∞(Γ\G). A functionϕis called pure ifϕ∈L2(Γ\G)(π) for someπ∈G.
For three pure smoothϕ1,ϕ2,ϕ3, the integral
Γ\Gϕ1(x)ϕ2(x)ϕ3(x)dxexists. Moreover,
fixπ1,π2,π3∈Gand fixG-equivariant embeddingsσj:Vπj→L
2(Γ\G), then
᐀aut
v1,v2,v3
=
Γ\Gσ1
v1
(x)σ2
v2
(x)σ3
v3
(x)dx (A.2)
defines aG-invariant continuous trilinear form onVπ∞1×Vπ∞2×Vπ∞3. LetL2(Γ\G)K be the space of allK-invariant vectors inL2(Γ\G). Then there is an orthonormal basis (ϕi)i∈
N
of pure vectors inL2(Γ\G)K. For eachi, letL
ibe theG-stable closed subspace ofL2(Γ\G) generated byϕi. Then the spacesLiare mutually orthogonal.
For fixed pure normalized ϕ,ϕ∈L2(Γ\G)K, we are interested in the growth of the sequence
cidef=᐀aut
ϕi,ϕ,ϕ. (A.3)
Let (λi,μ,ν) be the induction parameters of the representations given by (ϕi,ϕ,ϕ). First note that the numberscidecay exponentially in|λi|. So we define the renormalized se-quence
bi
def
=ci
2
expπλi. (A.4)
In [1], it is shown that ford=2, one has
|λi|≤T
bi≤ClogT, (A.5)
and in [8, Theorem 7.6], ford≥3,
|λi|≤T
bi≤CT2(d−2). (A.6)
Theorem A.1. Letd≥4. There isC >0 such that for everyT >0,
|λi|≤T
bi≤CT2(d−2). (A.7)
Proof. By the uniqueness of trilinear forms, there exists a constantaisuch that
᐀aut
ϕi,ϕ,ϕ=ai᐀st
ϕi,ϕ,ϕ. (A.8)
This constant depends on the embeddings ofπλ,πν,πμintoL2(Γ\G), but we can normal-ize these embeddings by insisting that the standard class-one vectorseλ,eμ,eνare mapped
toϕi,ϕ,ϕ. Thus the numberaionly depends onϕi,ϕ,ϕ.
Fix two smooth preunitaryG-representationsV,Vand letL,Lbe their Hilbert com-pletions. Consider the unitary representation ofG×GonL⊗Land letE=(L⊗L)∞be its smooth part. Denote byᏴ(E) the real vector space of all Hermitian forms onEand denote byᏴ+(E) the cone of nonnegative forms.
LetW be a smooth preunitary admissible representation ofG. AG-invariant func-tional᐀:W⊗V⊗V→Cdefines aG-equivariant linear mapl᐀:V⊗V→W∗which extends to aG-mapl᐀:E→W, whereWis the spaceWwith complex conjugate linear
structure. The Hermitian formHW onWinduces a formHW onW. Define a Hermitian formH᐀onEbyH᐀=l᐀∗HW, that is, foru,v∈E, one hasH᐀(u,v)=HW(l᐀(u),l᐀(v)). Finally, suppose thatW=Vλ,V,V are all principal series representations and let᐀st
be the standard trilinear form, then we writeHλstfor the Hermitian form induced onE. We also assume thatVλ,V,Vare cuspidal, that is, those fixed embeddings intoL2,∞are given. Then the cuspidal trilinear form᐀induces a Hermitian formHaut
λ onE.
Next consider the spaceL2,∞(Γ\G×Γ\G). LetHΔdenote the Hermitian form onL2,∞(Γ\
G×Γ\G) given by restriction to the diagonal, that is,
HΔ(u,v)=
Γ\Gu(x,x)v(x,x)dx. (A.9)
LetUbe a Hilbert space with Hermitian formH. For a closed subspaceLofU, letPrL denote the orthogonal projection fromU toLand letHLbe the Hermitian form onU given by
HL(u,v)=HPrL(u),PrL(v). (A.10)
The mapL→HLis additive and monotonic, that is, HL+L=HL+HL ifLandLare
orthogonal andHL≤HLifL⊂L.
Lemma A.2. On the spaceE, one has
i ai2
Hλsti≤HΔ. (A.11)
Proof. By the uniqueness, it follows thatHaut
λi = |ai|
2Hst
λi and since all the spacesLiare
The groupG×Gacts onL2,∞(Γ\G×Γ\G) and thus on the real vector space of Her-mitian forms onL2,∞(Γ\G×Γ\G). By integration, this action extends to a representation of the convolution algebraCc∞(G×G). Note that a nonnegative functionh∈C∞c (G×G) will preserve to cone of nonnegative Hermitian forms.
Lemma A.3. Leth∈Cc∞(G×G) be nonnegative. Then there existsC >0 such that
h·HΔ≤CHaut, (A.12)
whereHautis theL2-Hermitian form onL2,∞(Γ\G×Γ\G).
Proof. The formh·HΔ is given by integration over a smooth measure onΓ\G×Γ\G.
Being smooth, this measure is bounded by a multiple of the invariant measure.
A positive functional onᏴ(E) is an additive mapρ:Ᏼ+(E)→R≥
0∪ {∞}. For example,
every vectoru∈Egives rise to a positive functionalδudefined as
δu(H)=H(u,u). (A.13)
Every positive functional is monotonic and homogeneous, that is,
ρ(H)≤ρ(H) ifH≤H,
ρ(tH)=tρ(H) fort >0. (A.14)
Letρbe a positive functional and lethρ(λ)=ρ(Hλst). ThenLemma A.2implies that
i
hρ
λiai2≤ρ
HΔ. (A.15)
Proposition A.4. There are constantsT0,C >0 such that for everyT≥T0, there is a
posi-tive functionalρT onᏴ(E) with
ρTHΔ≤CTdim(M\K)+1, (A.16)
and withhT=hρTone hashT(λ)≥1/Cfor every|λ| ≤2T.
Proof. LetDbe a relatively compact open subset ofGwithg∈D1⇒g−1∈D. Assume
further thatDis invariant under right and left translations by elements of K. Let h∈
C∞c (G×G) such thathis constantly one onD×Dand such thathis invariant under left and right translates by elements ofK. The spaceEcan be identified withC∞(M\K×
M\K). Fix a norm · on the Lie algebra ofAN. Note that the map
ψ:A×N×M−→P\G×P\G, (a,n,m)−→w0anm,w0n0anm
(A.17)
is a diffeomorphism onto an open subset of (P\G)2∼=(M\K)2. LetF=ψ(1×1×M)=
theM-orbit of the point (w0,w0n0). LetC1,T >0 be large and set
S= (
ψ(a,n,m) :logan< 1
C1T,m∈M
)
LetuTbe a nonnegative smooth function on (M\K)2with support inSsuch that
u dx d y=1,
|u|2dx d y≤Tdim(M\K)+1. (A.19)
It is possible to construct such a function since the codimension ofFequals dim(M\K) + 1. With these data, define a positive functionalρT onᏴ(E) by
ρT(H)=δuT(h·H). (A.20)
Then one has
ρT
HΔ=ρu
h·HΔ≤Cρu
Hst
≤CTdim(M\K)+1. (A.21)
Recall the standard trilinear form ᐀st on Vλ⊗V⊗V and the corresponding lin-ear mapl᐀st:V⊗V→Vλ. We identify the spaceVλwithC∞(M\K) and letzdenote the pointM1∈M\K. The Dirac measureδz at z is a continuous linear functional on
C∞(M\K). We get an induced Hermitian formHzonEdefined by
Hz(u,v)=l᐀st(u)(z)l᐀st(v)(z). (A.22)
Lemma A.5. OnE, one has
Hst
λ =
K(k,k)·Hzdk. (A.23) Proof. This follows from the fact that the invariant Hermitian form onVλ is equal to
Kπ(k)Hzdk, whereHzis the Hermitian form onVλgiven byHz(u,v)=u(z)v(z). Since the test functionhwas assumed to beK×K-invariant, it follows that
hT(λ)=ρT
Hst
λ
=δu
h·Hz
. (A.24)
So in order to establish a lower bound for ρT(Hλst), it suffices to give a lower bound for δu(x·Hz)= |π(x)fz,u|2 forx∈D, where f
z is the linear functional onEgiven by fz(u)=δz(l᐀st(u)).
Lemma A.6. There isC >0 such that forT0 large enough, there exists an open, nonempty
subsetD0⊂Dsuch that forT≥T0,|λ| ≤T, andx∈D0,|π(x)fz,uT| ≥1/C. Proof. Note first that withx=ank∈G, we have
*
π(x)fz,uT +
=
(M\K)2uT
k2,k3
φk,kank
2 ,k3ank
dk2dk3. (A.25)
We next derive an explicit formula forφon the open orbit. Recall that
᐀st(f)=
Gf
y,w)y,w0n0y
d y
=
ANKf
ank,w0ank,w0n0ank
da dn dk,
and that the last integrand equals
aλ+ρaw
0an
μ+ρ
aw0n0an
ν+ρ
fk,kw0an
k,kw0n0an
kda dn dk. (A.27)
We deduce that
φk,kw0ank,kw0n0ank=aλ+ρaw0anμ+ρaw0n0anν+ρW(an), (A.28)
whereWis the Radon-Nykodim derivative ofda dn dkagainst the measure on (M\K)3.
In particular, this function is positive and smooth and does not depend onλ. It follows that the functionalπ(x)fzis given by integration against the function
π(ank)fz
w0anm,w0n0anm
=φk,w0aanmk,w0n0aanmk
, (A.29)
wheren=a−1namn(m)−1. Note thatMk=Mmkand so the above equals
(aa)λ+ρaw0aan
μ+ρ
aw0n0aan
ν+ρ
W(aan). (A.30)
Note in particular that the functionπ(x)fzis invariant underM.
The setSis a tubular neighborhood of the compact setF. We have natural local co-ordinates withF-directions, which can be identified with open parts of M-orbits and
F⊥-directions, which can be viewed asAN-orbits. The above formula shows that the gra-dient ofπ(x)fzis zero along theM-orbits. Along theAN-orbits, this gradient is bounded by a constant timesT, if we leaveμandνfixed and restrict to|λ| ≤T. LetC0>0 be large
and letD0⊂Dbe the subset of elementsx∈Dsuch that|π(x)fz|< C0. This set is open
and nonempty ifC0is large enough and it does not depend onλ.
Lemma A.6impliesProposition A.4and the latter impliesTheorem A.1as follows. Consider the inequalityi|ai|2ρT(Hλsti)≤ρT(HΔ). ByProposition A.4, the right-hand
side is bounded byCT2 dim(M\K)and since dimM\K=dimP\G=dimN=d−1, we get
i ai2
ρT
Hst
λi
≤CTd. (A.31)
Restricting the sum to thoseiwith|λi| ≤2Tand using the second assertion ofProposition A.4, we get
|λi|≤T
ai2
≤CTd. (A.32)
According toTheorem 4.1, we havebi|λi|4−d∼|ai|2, and so
|λi|≤T
biλi
4−d≤CTd. (A.33)
This implies the theorem.
Acknowledgment
References
[1] J. Bernstein and A. Reznikov, Estimates of automorphic functions, Moscow Mathematical Journal 4 (2004), no. 1, 19–37, 310.
[2] M. Harris and S. S. Kudla, The central critical value of a triple product L-function, Annals of
Mathematics. Second Series 133 (1991), no. 3, 605–672.
[3] , On a conjecture of Jacquet, Contributions to Automorphic Forms, Geometry, and Num-ber Theory, Johns Hopkins University Press, Maryland, 2004, pp. 355–371.
[4] M. Harris and A. J. Scholl, A note on trilinear forms for reducible representations and Beilinson’s
conjectures, Journal of the European Mathematical Society 3 (2001), no. 1, 93–104.
[5] S. Helgason, Differential Geometry, Lie Groups, and Symmetric Spaces, Pure and Applied
Mathe-matics, vol. 80, Academic Press, New York, 1978.
[6] D. Jiang, Nonvanishing of the central critical value of the triple productL-functions, International
Mathematics Research Notices 1998 (1998), no. 2, 73–84.
[7] , On Jacquet’s conjecture: the split period case, International Mathematics Research No-tices 2001 (2001), no. 3, 145–163.
[8] B. Kr¨otz and R. J. Stanton, Holomorphic extensions of representations. I. Automorphic functions, Annals of Mathematics. Second Series 159 (2004), no. 2, 641–724.
[9] H. Y. Loke, Trilinear forms ofgl2, Pacific Journal of Mathematics 197 (2001), no. 1, 119–144.
[10] V. F. Molˇcanov, Tensor products of unitary representations of the three-dimensional Lorentz group, Izvestiya Akademii Nauk SSSR. Seriya Matematicheskaya 43 (1979), no. 4, 860–891, 967. [11] A. I. Oksak, Trilinear Lorentz invariant forms, Communications in Mathematical Physics 29
(1973), no. 3, 189–217.
Anton Deitmar: Mathematisches Institut, Eberhard Karls Universit¨at T¨ubingen, Auf der Morgenstelle 10, T¨ubingen 72076, Germany