2.5 Hilbert Spaces
2.5.2 Hilbert Spaces: Their Coordinatizations
where we used the Cauchy-Schwarz inequality.
ii. Recall that
0 ≤ (kfk − kgk)2 =kfk2+kgk2−2kfk kgk. (2.6) iii. Adding this non-negative quantity to the r.h.s. of (2.5) in- creases it. Consequently, Eqs.(2.5) and (2.6) ⇒ kf +gk2 ≤
2kfk2+ 2kgk2.
Thus we have
f, g square integrable⇒f +g is square integrable, i.e. L2 is indeed closed under addition.
(c) L2(a, b) is Cauchy complete.
2.5.2
Hilbert Spaces: Their Coordinatizations
Lecture 14
The importance ofL2derives from the fact that its elements refer, among oth-
ers, to the finite energy states of an archetypical system, a vibrating system or the finite energy signals of an ensemble of messages. These states/signals are mathematized in terms of functions. On the other hand, for the purpose of measurement they need to be, or have been, represented by sequences of numbers, i.e. by elements of ℓ2. Without measurement data expressed
in terms of these elements, these states would be mere floating abstraction disconnected from the physical world. It is the elements of ℓ2 which ground
such finite energy states in observations and measurements. Granted their epistemic foundation in the physical world, what is the role of L2 and ℓ2 in
To this end recall that, given an n-dimensional vector space V, then a choice of basis determines an isomorphism F which relates V to Rn, the
space of n-tuples, a coordinate realization of V: V −−−−−−→F Rn u ∼∼∼∼❀ F(u) = c1 ... cn Here
(i) F is induced by a given system of orthonormal spanning vectors uk.
This means that once {uk} is given, F is determined: for any u in V
F yields a unique F(u), the array expansion coefficients ck.
(ii) F is linear. This means thatF is mathematized by means of a matrix. (iii) F is one-to-one.
(iv) F is onto,
where “onto” means that, given anyv ∈Rn, one can solve
F(u) =v (2.7)
foru∈V, while “one-to-one” means that such a solution is unique. In brief, F is an isomorphic relation betweenV and Rn, and it is induced by{u
k}nk=1.
The extension of the idea of such an isomorphism to infinite-dimensional Hilbert spaces results in the claim that a Hilbert spaceH has ℓ2 asits coor-
dinate realization. Relating these two spaces is the isomorphism F, H −−−−−−→F ℓ2
f ∼∼∼∼❀ F[f] = {c1, c2,· · · }
Here, as in the finite dimensional case,
(i) F is induced by a given system of orthonormal vectors uk.
(ii) F is linear. (iii) F is one-to-one.
2.5. HILBERT SPACES 77
(iv) F is onto.
An isomorphism is a two-way map H −→ ℓ2 and H ←−ℓ2. In finite dimen-
sions its validation is achieved by algebraic manipulations on a basis-induced system of equations.
In infinite dimensions, however, the method for validation is necessarily different. It is a three-step process:
I. Identify an infinite system of o.n. vectors. This sytem induces a linear map F fromH to ℓ2.
II. Apply Bessel’s least squares theorem to the system of o.n. vectors. This application starts by constructing the linear map,
F : H −→ℓ2 . (2.8)
The map F is the unifying kingpin in the whole subsequent devel- opment, as summarized by the three theorems 1.5.1, 1.5.2, and 1.5.3 below.
According to Bessel’s least squares theorem (1.5.1 below), even if more than one f ∈ H yields the same {ck} ∈ ℓ2, each one of them has
the same optimal (least squared error) representative in the subspace spanned by those o.n. vectors. This representative is the result of Bessel’s least squares error analysis. If the least squared error is zero, i.e. Parseval’ identity is satisfied, Bessel’s theorem guarantees that there is only a single function f having the particular least squares representation induced by the system of o.n. vectors. In other words, the map F is one-to-one (Theorem 1.5.2).
III. Use the Riesz-Fischer Theorem (1.5.3). Its gist is the fundamental feature that, for a system with an infinite number of o.n. vectors, this map is always onto. In other words, for every element {ck} ∈ ℓ2 there
is at least one element f ∈ H.
In summary, Fischer and Riesz guarantee that F is onto, while Bessel and Parseval guarantee that it is one-to-one. In brief, F is an isomorphism. Put differently, as exemplified by the contributions of these four workers, human knowledge of mathematical methods is not a mere collection, but a
structure. Their works, even though developed separately, does not amount to a juxtaposition. By applying the concept of an isomorphism to their works one forms a unified structure, the coordinatization of a Hilbert space.
Bessel and Parceval
Bessel’s fundamental optimization process, in particular the construction of of the linear map Eq.(2.8), is summarized by the following
Theorem 2.5.1 (Bessel)
(Least squares approximation via subspaces) Given an o.n. system u1, u2, . . . , uk, . . .
in the Hilbert space H ⊆L2, let f be an arbitrary element ofH. Then
(i) the expression
f− N X k=1 akuk 2 ≡EN2(a1, . . . , aN)
has a minimum, for
ak =huk, fi ≡ck k = 1,· · · , N .
(ii) This minimum equals
kfk2−
N
X
k=1
|ck|2 =EN2(c1, . . . , cN) N = 1,2,· · · .
with its associated hierarchy kfk2 ≥E2 1(c1)≥E22(c1, c2)≥ · · · ≥EN2(c1,· · · , cN)≥ · · · ≥0 . (iii) Moreover, ∞ X k=1 |ck|2 ≤ kfk2,
a result known as Bessel’s inequality.
Note: This theorem introduces en passant two new concepts which are key to the subsequent development:
2.5. HILBERT SPACES 79
• The coefficients
ck =huk, fi
are called the (generalized) Fourier coefficients. They form the image of the function f under the linear transformation
F : H →ℓ2 (2.9)
f ∼❀F[f] ={ck}∞k=1 (2.9′) • The sequence of sums
N
X
k=1
ckuk ≡SN N = 1,2,· · · ,
is called the sequence of partial Fourier series of f, withSN being the
Nth partial Fourier series.
Nota bene: The function E2
N(a1, . . . , aN) is called Gauss’s mean squared er-
ror. Minimizing it by setting ∂EN2
∂ak
= 0 k = 1, . . . , N yields the N Fourier coefficients
ak=huk, fi ≡ck k= 1, . . . , N
as thesolution to this equation (try it!). The word “mean” in Gauss’s mean squared error arises from its defining property,
EN2 = Z b a | f(x)− N X k=1 akuk(x)|2ρ(x)dx .
The integrand |f(x)−PNk=1akuk(x)|2 is the error at x, while the integral is
(b−a) times the (weighted) “mean” of this quantity, in compliance with the mean value theorem of integral calculus.
Proof: The Gaussian mean squared error function hf− N X k=1 akuk, f − N X ℓ=1 aℓuℓi ≡EN2
is a quadratic expression in the complex unknowns ak. As usual, in such
expressionscompleting the square will yield the minimum value at a glance. Multiplying out the inner product yields
E2 n=kfk2− N X k=1 akhuk, fi − N X ℓ=1 aℓhf, uℓi+ N X k=1 N X ℓ=1 akaℓhuk, uℓi.
By (i) introducing the Fourier coefficients ck =huk, fi
of f relative to the system {uk}, (ii) using the orthonormality of the uk’s
yields, and (iii) adding and subtracting
N P k | ck|2, one obtains E2 N = kfk2 − PN k=1akck − PNℓ=1aℓcℓ + PNk=1akak − PNk=1|ck|2 + PN k=1ckck = kfk2 − PN k=1|ck|2 + PN k=1|ak−ck|2
This expression is the key to validating the three conclusions of the theorem. (i) E2
N achieves its minimum when
ak =ck .
ThusF[f] ={ck}∞k=1 islinear.
(ii) The minimum value of E2
N is EN2(c1,· · · , cN) = kf−SNk2 (2.10) =kfk2 − N X k=1 |ck|2 .
(iii) The fact that this holds for all integers N implies ∞
X
k=1
|ck|2 ≤ kfk2 ,
2.5. HILBERT SPACES 81
Bessel’s Inequality: Its Geometrical Meaning
This theorem can also be summarized geometrically as follows: 1. The set of linear combinations
span{u1, . . . , uN} ≡WN ⊂ H ⊆L2
is a subspace of L2, and the Nth partial Fourier sum
N
X
k=1
ckuk≡w∗N (2.11)
is the orthogonal projection of f onto WN. The squared length of wN∗
is kw∗Nk2 =h N X k=1 ckuk, N X k=1 ckuki = N X k=1 |ck|2 ,
which is the Pythagorean theorem in WN.
2. This projection of f onto WN is linear. It is given by
w∗N =
N
X
k=1
ukhuk, fi ≡PWNf (∈WN) . (2.12)
It is depicted in Figure 2.6, and it has the property that PWNPWNf =PWNf for all f ∈ H .
This expresses the fact thatPWN is the identity operator onWN. On the
other hand, in light of Bessel’s inequality, PWN shortens f iff 6∈WN.
3. The triangle formed by f,w∗
N ∈WN, and the error vector
h∗N =f −wN∗ ; wN∗ =
N
X
k=1
W = span{u , ... , u }N
1
N
f
h*N
u1
u2
w*N
Figure 2.6: The N-dimensional subspace WN of the ambient Hilbert space
H = L2. The least squares approximation w∗
N is the orthogonal projection
of the vector f ontoWN. The difference between the given vector f and its
projection onto the subspace is the error vector h∗
N.
is a right triangle: the sides of the triangle obey the Pythagorean the- orem in the ambient Hilbert space:
kh∗Nk2 =kf − N X k=1 ckukk2 (2.13) =hf − N X k=1 ckuk, f− N X k=1 ckuki =kfk2− N X k=1 |ck|2 (2.14) =kfk2− kw∗ Nk2
This evidently yields
kfk2 ≥ kw∗
Nk2
which is the finite-dimensional version of Bessel’s inequality. Using the optimal (= “least square”) approximation
wN∗ =
N
X
i=1
2.5. HILBERT SPACES 83 one obtains kfk2 ≥ N X i=1
ckck; ckdetermined by the least squares approximation,
which for our square integrable functions is
Z b a | f(x)|2ρ(x)dx≥ N X i=1 |ck|2.
Geometrically this inequality says (length of vector)2 ≥
length of its projection onto the subspace WN
2
.
Consider now a sequence of subspaces
W1 ⊂W2 ⊂ · · · ⊂WN ⊂WN+1 ⊆ · · ·,
the respective optimal approximations to the given function f, and the corresponding sequence of least square errors
kh∗Nk2 =kf −
N
X
i=1
ckukk2 , N = 1,2, . . . .
This sequence not only reveals the quality of each partial sum approxi- mation. Ifkh∗
Nk2 approaches zero asN tends to infinity, then this very
fact also reveals something about {uk}. Indeed, whenever kh∗Nk2 →0,
the o.n. system is one which constitutes a basis for H, meaning that Eq.(2.15) or (2.17) is satisfied.
4. With kh∗
Nk2 as the shortest squared distance between f and WN, the
error vector h∗
N is perpendicular to WN:
The Orthonormal System
Highlighting the fact that any o.n. system induces a linear transformation intoℓ2 is only the first step in using a Hilbert space to conceptualize the mea-
sured properties of things in science. Is the chosen o.n. system appropriate for this task? There are many kinds of o.n. systems and their concomi- tant linear transformations. The necessity of grounding a Hilbert space in the measurements and observations of the physical world requires answers to two key questions about any o.n. system and its linear transformations:
1. Is it onto (“surjective”)? 2. Is it one-to-one (“injective”)? Their answers require the following Definition.
A system of o.n. vectors {uk:k = 1,2,· · · } is said to be closed whenever
∞
X
k=1
|ck|2 =kfk2 (2.15)
for every f in H.
Under such a circumstance one refers to this relation as Parseval’s iden- tity.1
By taking the limit as N → ∞ of the right hand sides of Eqs.(2.13) and (2.14), one obtains the result that
∞ X k=1 |ck|2 =kfk2 ⇐⇒ lim N→∞kf− N X k=1 ckukk2 = 0 (2.16)
wheneverck =huk, fi. This equivalence is the mathematization of the portal
between any Hilbert space and its coordinate realization by an “appropriate” system of o.n. vectors. That the two equalities imply each other is the result of a mere algebraic evaluation of the involved inner products.
Exercise 2.5.1 (PARSEVAL’S IDENTITY AND FOURIER SERIES)
Let
ck≡ huk, fi k= 1,2,· · ·
2.5. HILBERT SPACES 85
be the Fourier coefficients off ∈L2 relative to the orthonormal system{u
k}∞k=1⊂ L2. Prove: ∞ X k=1 |ck|2=kfk2⇐⇒ kf− ∞ X k=1 ckukk2 = 0 .
The concept “appropriate” is too broad and not particular enough to identify the kind of system worthy of thorough study. This deficiency is remedied by the concept having the following
Definition
A system of o.n. vectors {uk}∞k=1 is said to be complete whenever
lim N→∞kf − N X k=1 ckukk2 = 0 (2.17) i.e. f=˙ ∞ X k=1 ckuk Comment
This is to be compared with pointwise equality, which is expressed by the statement that f(x) = ∞ X k=1 ckuk(x) . (2.18)
The difference between “=” and “=” manifests itself only when the square-. summable sequence {ck}yields a function which has one or more discontinu-
ities, then one does not have pointwise equality, Eq.(2.18). Instead, one has the weaker condition, Eq.(2.17). This condition does not specify the value of f at the point(s) of discontinuity. Instead, it specifies anequivalence class of functions, all having the same graph everywhere except at the point(s) of discontinuity.
Completeness Relation For H ⊆ L2(a, b)
An “appropriate” system is one for which either equation in Eq.(2.16), i.e. ∞ X k=1 hf, ukihuk, fi=hf, fi ⇐⇒f=˙ ∞ X k=1 ukhuk, fi (2.19)
holds for all f ∈ H. Moreover, the statement hf, fi= ∞ X k=1 hf, ukihuk, fi ∀f ∈ H
implies and is implied by hf, gi=
∞
X
k=1
hf, ukihuk, gi ∀f and g ∈ H.
Explicitly, one has
Z b a f(x)g(x)ρ(x)dx= ∞ X k=1 Z b a f(x)uk(x)ρ(x)dx Z b a uk(x′)g(x′)ρ(x′)dx′.
This can be rewritten in terms of the Dirac delta function (which is developed in Section 3.2 starting on page 135) as
Z b a Z b a f(x)δ(x−x′)g(x′)ρ(x)dxdx′ = Z b a Z b a f(x) ∞ X k=1 uk(x)uk(x′)ρ(x)ρ(x′)g(x′)dxdx′.
This holds for all f, g ∈ H = L2(a, b). Consequently, we have the
following alternate form for the completeness of the set of orthonormal functions δ(x−x′) ρ(x′) = ∞ X k=1 uk(x)uk(x′) or δ(x−x′) ρ(x′) = ∞ X k=1 |uk(x)ihuk(x′)|
in quantum mechanical notation.
Usually the orthonormal functions uk are the eigenfunctions of some
operator (for example, the Sturm-Liouville operator + boundary con- ditions, which we have met in chapter 1 on page 20). The Dirac delta function
δ(x−x′) ρ(x′) =
δ(x−x′) ρ(x)
2.5. HILBERT SPACES 87
is the identity operator on the Hilbert space H. Consequently, the alternate form of the completeness relation
δ(x−x′) ρ(x) = ∞ X k=1 uk(x)uk(x′) (2.20)
can be viewed as a spectral representation of the identity operator in H.
Thus thecompleteness of a set{uk}∞k=1 refers to the fact that it contains
sufficiently many uk’s of the right kind so that the identity transformation
(2.20) can be represented in terms of them. Equivalently, the uk’s form a
spanning set for H ⊆L2.
On the other hand, in the quest for alternative mathematical precision, some mathematicians asked and answered the following important question: Is an o.n. system{uk}∞k=1unique whenever it gives rise to Parseval’s identity?
In other words, can such a {uk} be a proper subset of any other orthonor-
mal set in H? The reason that the answer is “no” is that they call {uk} a
maximal o.n. sequence.
Band-Limited L2 Message Spaces
Example (Complete vs. incomplete system of o.n. band-limited L2 signal
functions.)
Consider the following three Hilbert spaces: H[0,ε]= f ∈L2(−∞,∞) : Z ∞ −∞ e−iωx √ 2πf(x)dx= 0; ω6∈[0, ε] , (2.21) H[ε,2ε]= g ∈L2(−∞,∞) : Z ∞ −∞ e−iωx √ 2πg(x)dx= 0; ω6∈[ε,2ε] , (2.22) H[0,2ε]= h∈L2(−∞,∞) : Z ∞ −∞ e−iωx √ 2πh(x)dx= 0; ω 6∈[0,2ε] (2.23) In mathematical engineering each one refers to a set of band-limited signals: The Fourier amplitudes of the f’s, g’s, and h’s are non-zero only in the frequency windows [0, ε], [ε,2ε], and [0,2ε] respectively.
Each of these spaces is a type of function space. Indeed, each of the f’s, g’s, and h’s is a particular signal function. The difference between the three
is the difference in how a particular signal function is put into mathematical form (“mathematized”). Although there are many ways of doing this, here we shall do it by means of Fourier series: Consider the following three systems of o.n. vectors:
{uk(x)}={P0εk(x) : k = 0,±1,· · · } (2.24)
{uk′(x)}={P1εk′(x) : k′ = 0,±1,· · · } (2.25)
{uk′′(x)}={P02kε′′(x) : k′′= 0,±1,· · · }. (2.26)
Borrowing from Section 3.4.2 the o.n. wave packets Pjℓε(x) = Z (j+1)ε jε e−2πiℓω/ε √ ε eiωx √ 2πdω , (2.27)