17. Inner product spaces Definition Let V be a real vector space. An inner product on V is a function

17. Inner product spaces

Definition 17.1. Let V be a real vector space. An inner product on V is a function

h , i : V × V −→ R, which is

• symmetric, that is

hu, vi = hv, ui.

• bilinear, that is linear (in both factors):

hλu, vi = λhu, vi, for all scalars λ and

hu

+ u

, vi = hu

, vi + hu

, vi, for all vectors u

, u

and v.

• positive that is

hv, vi ≥ 0.

• non-degenerate that is if hu, vi = 0 for every v ∈ V then u = 0.

We say that V is a real inner product space. The associated quadratic form is the function

Q : V −→ R, defined by

Q(v) = hv, vi.

Example 17.2. Let A ∈ M

(R) be a real matrix. We can define a function

h , i : R

× R

−→ R, by the rule

hu, vi = u

Av.

The basic rules of matrix multiplication imply that this function is bi- linear. Note that the entries of A are given by

a

= e

Ae

= he

, e

i.

In particular, it is symmetric if and only if A is symmetric that is A

= A. It is non-degenerate if and only if A is invertible, that is A has rank n. Positivity is a little harder to characterise.

Perhaps the canonical example is to take A = I

. In this case if u = (r

, r

, . . . , r

) and v = (s

, s

, . . . , s

) then u

I

v = P r

s

. Note

1

that if we take u = v then we get P r

. The square root of this is the Euclidean distance.

Definition 17.3. Let V be a real vector space. A norm on V is a function

k.k : V −→ R, what has the following properties

•

kkvk = |k|kvk, for all vectors v and scalars k.

• positive that is

kvk ≥ 0.

• non-degenerate that is if kvk = 0 then v = 0.

• satisfies the triangle inequality, that is ku + vk ≤ kuk + kvk.

Lemma 17.4. Let V be a real inner product space.

Then

k.k : V −→ R, defined by

kvk = phv, vi, is a norm on V .

Proof. It is clear that the norm satisfies the first property and that it is positive. Suppose that u ∈ V . By assumption there is a vector v such that

hu, vi 6= 0.

Consider

0 ≤ hu + tv, u + tvi

= hu, ui + 2thu, vi + t

hv, vi.

If hv, vi = 0 then certainly hu, ui > 0. Otherwise put t = − hu, vi

hv, vi > 0.

Then

hu, ui + 2thu, vi + t

hv, vi = hu, ui + thu, vi.

Once again

hu, ui > (hu, vi)

hv, vi > 0.

Thus the norm is non-degenerate.

Now suppose that u and v ∈ V . Then

hu + v, u + vi = hu, ui + 2hu, vi + hv, vi

≤ kuk

+ 2kuk · kvk + kvk

= (kuk + kvk)

.

Taking square roots gives the triangle inequality.  Note that one can recover the inner product from the norm, using the formula

2hu, vi = Q(u + v) − Q(u) − Q(v),

where Q is the associated quadratic form. Note the annoying ap- pearence of the factor of 2.

Notice also that on the way we proved:

Lemma 17.5 (Cauchy-Schwarz-Bunjakowski). Let V be a real inner product space.

If u and v ∈ V then

hu, vi ≤ kuk · kvk.

Definition 17.6. Let V be a real vector space with an inner product.

We say that two vectors v and w are orthogonal if hu, vi = 0.

We say that a basis v

, v

, . . . , v

is an orthogonal basis if the vectors v

, v

, . . . , v

are pairwise orthogonal. If in addition the vectors v

have length one, we say that v

, v

, . . . , v

is an orthonormal basis.

Lemma 17.7. Let V be a real inner product space.

(1) If the vectors v

, v

, . . . , v

are pairwise orthogonal then they are independent. In particular if m = dim V then v

, v

, . . . , v

are an orthogonal basis of V .

(2) If v

, v

, . . . , v

are an orthonormal basis of V and v ∈ V then v = X

r

v

, where

r

= hv, v

i.

3

Proof. We first prove (1). Suppose that

r

v

+ r

v

+ · · · + r

v

= 0.

Taking the inner product of both sides with v

gives 0 = hr

v

+ r

v

+ · · · + r

v

, v

i

=

m

X

i=1

r

hv

, v

i

= r

hv

, v

i.

As

hv

, v

i 6= 0, it follows that r

= 0. This is (1).

The proof of (2) is similar. 

So how does one find an orthonormal basis?

Algorithm 17.8 (Gram-Schmidt). Let v

, v

, . . . , v

be independent vectors in a real inner product space V .

(1) Let 1 ≤ k ≤ n be the largest index such that v

, v

, . . . , v

are orthonormal.

(2) If k = m then stop.

(3) Otherwise let

u

= v

− r

v

− r

v

− · · · − r

v

, where r

= hv

, v

i. Replace v

by

u

ku

k , and return to (1).

In practice the algorithm works as follows. First we replace v

by v

kv

k ,

so that v

has unit length. Then we consider v

. We have to subtract some of v

to ensure that it is orthogonal to v

. So consider a vector of the form

u = v

+ λv

,

where λ is chosen to make u orthogonal to v

. We have 0 = hu, v

i = hv

, v

i + λhv

, v

i, so that

λ = −hv

, v

i.

Then we rescale to get a vector of unit length. At the next stage, we can choose λ and µ so that

v

+ λv

+ µv

,

is orthogonal v

and v

. The key thing is that since v

and v

are orthogonal, our choice of λ and µ are independent of each other.

For example, consider the vectors

v

= (1, −1, 1), v

= (1, 0, 1) and v

= (1, 1, 2), in R

with the usual inner product. The first step is to replace v

by

v

= 1

√ 3 (1, −1, 1).

Now let

u = (1, 0, 1) − 2

3 (1, −1, 1)

= 1

3 (1, 2, 1).

Then we replace u by a vector parallel to u of unit length v

= 1

√ 6 (1, 2, 1).

Finally we put

u = (1, 1, 2) − 2

3 (1, −1, 1) − 5

6 (1, 2, 1)

= 1

2 (−1, 0, 1).

Finally we replace u by a vector of unit length, v

= 1

√ 2 (−1, 0, 1).

Thus v

= 1

√ 3 (1, −1, 1) v

= 1

√ 6 (1, 2, 1) and v

= 1

√ 2 (−1, 0, 1), is the orthonormal basis produced by Gram-Schmidt.

One very useful property of inner products is that we get canonically defined complimentary linear subspaces:

Lemma 17.9. Let V be a finite dimensional real inner product space.

If U ⊂ V is a linear subspace, then let

U

= { w ∈ V | hw, ui = 0, ∀u ∈ U },

the set of all vectors orthogonal to every element of U . Then

5

• U

is a linear subspace of V .

• U ∩ U

= {0}.

• U and U

span V .

In particular V is isomorphic to U ⊕ U

.

Proof. We first prove (1). Suppose that w

and w

∈ U

. Pick u ∈ U . Then

hw

+ w

, ui = hw

, ui + hw

, ui

= 0 + 0 = 0.

It follows that w

+ w

∈ U

and so U

is closed under addition. Now suppose that w ∈ U

and λ is a scalar. Then

hλw, ui = λhw, ui

= λ0 = 0.

Thus λw ∈ U

and so U

is closed under scalar multiplication. Thus U

is a linear subspace of V . This is (1).

Suppose that w ∈ U ∩ U

. Then hw, wi = 0.

But then w = 0. This is (2).

Suppose that v ∈ V . If v ∈ U there is nothing to prove. Otherwise pick an orthonormal basis u

, u

, . . . , u

of U . Then the vectors v, u

, u

, . . . , u

are independent. By Gram-Schmidt we may find scalars r

, r

, . . . , r

such that

w = v − X r

u

,

is orthogonal to u

, u

, . . . , u

(in fact r

= hv, u

i). But then w is orthogonal to U , that is w ∈ U

. Let u = P r

u

∈ U . Then v = u+w.

This is (3).