8Inner product spaces

IB Linear Algebra

8.2 Gram-Schmidt orthogonalization

As mentioned, we want to make sure every vector space has an orthonormal

basis, and we can extend any orthonormal set to an orthonormal basis, at least

in the case of finite-dimensional vector spaces. The idea is to start with an

arbitrary basis, which we know exists, and produce an orthonormal basis out of

it. The way to do this is the Gram-Schmidt process.

Theorem

(Gram-Schmidt process)

.

Let

V

be an inner product space and

e

1

, e

2

, ···

a linearly independent set. Then we can construct an orthonormal set

v

1

, v

2

, ··· with the property that

hv

1

, ··· , v

k

i = he

1

, ··· , e

k

i

for every k.

Note that we are not requiring the set to be finite. We are just requiring it

to be countable.

Proof.

We construct it iteratively, and prove this by induction on

k

. The base

case k = 0 is contentless.

Suppose we have already found v

1

, ··· , v

k

that satisfies the properties. We

define

u

k+1

= e

k+1

−

k

X

i=1

(v

i

, e

i+1

)v

i

.

We want to prove that this is orthogonal to all the other

v

i

’s for

i ≤ k

. We have

(v

j

, u

k+1

) = (v

j

, e

k+1

) −

k

X

i=1

(v

i

, e

k+1

)δ

ij

= (v

j

, e

k+1

) − (v

j

, e

k+1

) = 0.

So it is orthogonal.

We want to argue that u

k+1

is non-zero. Note that

hv

1

, ··· , v

k

, u

k+1

i = hv

1

, ··· , v

k

, e

k+1

i

since we can recover

e

k+1

from

v

1

, ··· , v

k

and

u

k+1

by construction. We also

know

hv

1

, ··· , v

k

, e

k+1

i = he

1

, ··· , e

k

, e

k+1

i

by assumption. We know

he

1

, ··· , e

k

, e

k+1

i

has dimension

k

+ 1 since the

e

i

are

linearly independent. So we must have

u

k+1

non-zero, or else

hv

1

, ··· , v

k

i

will

be a set of size

k

spanning a space of dimension

k

+ 1, which is clearly nonsense.

Therefore, we can define

v

k+1

=

u

k+1

ku

k+1

k

.

Then

v

1

, ··· , v

k+1

is orthonormal and

hv

1

, ··· , v

k+1

i

=

he

1

, ··· , e

k+1

i

as re-

quired.

Corollary.

If

V

is a finite-dimensional inner product space, then any orthonor-

mal set can be extended to an orthonormal basis.

Proof.

Let

v

1

, ··· , v

k

be an orthonormal set. Since this is linearly independent,

we can extend it to a basis (v

1

, ··· , v

k

, x

k+1

, ··· , x

n

).

We now apply the Gram-Schmidt process to this basis to get an orthonormal

basis of

V

, say (

u

1

, ··· , u

n

). Moreover, we can check that the process does not

modify our v

1

, ··· , v

k

, i.e. u

i

= v

i

for 1 ≤ i ≤ k. So done.

Definition

(Orthogonal internal direct sum)

.

Let

V

be an inner product space

and

V

1

, V

2

≤ V

. Then

V

is the orthogonal internal direct sum of

V

1

and

V

2

if it

is a direct sum and V

1

and V

2

are orthogonal. More precisely, we require

(i) V = V

1

+ V

2

(ii) V

1

∩ V

2

= 0

(iii) (v

1

, v

2

) = 0 for all v

1

∈ V

1

and v

2

∈ V

2

.

Note that condition (iii) implies (ii), but we write it for the sake of explicitness.

We write V = V

1

⊥ V

2

.

Definition

(Orthogonal complement)

.

If

W ≤ V

is a subspace of an inner

product space V , then the orthogonal complement of W in V is the subspace

W

⊥

= {v ∈ V : (v, w ) = 0, ∀w ∈ W }.

It is true that the orthogonal complement is a complement and orthogonal,

i.e. V is the orthogonal direct sum of W and W

⊥

.

Proposition.

Let

V

be a finite-dimensional inner product space, and

W ≤ V

.

Then

V = W ⊥ W

⊥

.

Proof.

There are three things to prove, and we know (iii) implies (ii). Also, (iii)

is obvious by definition of W

⊥

. So it remains to prove (i), i.e. V = W + W

⊥

.

Let w

1

, ··· , w

k

be an orthonormal basis for W , and pick v ∈ V . Now let

w =

k

X

i=1

(w

i

, v)w

i

.

Clearly, we have

w ∈ W

. So we need to show

v − w ∈ W

⊥

. For each

j

, we can

compute

(w

j

, v − w) = (w

j

, v) −

k

X

i=1

(w

i

, v)(w

j

, w

i

)

= (w

j

, v) −

k

X

i=1

(w

i

, v)δ

ij

= 0.

Hence for any λ

i

, we have

X

λ

j

w

j

, v − w

= 0.

So we have v − w ∈ W

⊥

. So done.

Definition

(Orthogonal external direct sum)

.

Let

V

1

, V

2

be inner product spaces.

The orthogonal external direct sum of

V

1

and

V

2

is the vector space

V

1

⊕V

2

with

the inner product defined by

(v

1

+ v

2

, w

1

+ w

2

) = (v

1

, w

1

) + (v

2

, w

2

),

with v

1

, w

1

∈ V

1

, v

2

, w

2

∈ V

2

.

Here we write v

1

+ v

2

∈ V

1

⊕ V

2

instead of (v

1

, v

2

) to avoid confusion.

This external direct sum is equivalent to the internal direct sum of

{

(

v

1

, 0

) :

v

1

∈ V

1

} and {(0, v

2

) : v

2

∈ V

2

}.

Proposition.

Let

V

be a finite-dimensional inner product space and

W ≤ V

.

Let (

e

1

, ··· , e

k

) be an orthonormal basis of

W

. Let

π

be the orthonormal

projection of

V

onto

W

, i.e.

π

:

V → W

is a function that satisfies

ker π

=

W

⊥

,

π|

W

= id. Then

(i) π is given by the formula

π(v) =

k

X

i=1

(e

i

, v)e

i

.

(ii) For all v ∈ V, w ∈ W , we have

kv − π(v)k ≤ kv − wk,

with equality if and only if

π

(

v

) =

w

. This says

π

(

v

) is the point on

W

that is closest to v.

W

⊥

w

v

π(v)

Proof.

(i) Let v ∈ V , and define

w =

X

i=1

(e

i

, v)e

i

.

We want to show this is

π

(

v

). We need to show

v − w ∈ W

⊥

. We can

compute

(e

j

, v − w) = (e

j

, v) −

k

X

i=1

(e

i

, v)(e

j

, e

i

) = 0.

So v − w is orthogonal to every basis vector in w, i.e. v − w ∈ W

⊥

.So

π(v) = π(w) + π(v − w) = w

as required.

(ii)

This is just Pythagoras’ theorem. Note that if

x

and

y

are orthogonal,

then

kx + yk

2

= (x + y, x + y)

= (x, x) + (x, y) + (y, x) + (y.y)

= kxk

2

+ kyk

2

.

We apply this to our projection. For any w ∈ W , we have

kv − wk

2

= kv − π(v)k

2

+ kπ(v) − wk

2

≥ kv − π(v)k

2

with equality if and only if kπ(v) − wk = 0, i.e. π(v) = w.