8Inner product spaces
IB Linear Algebra
8.2 Gram-Schmidt orthogonalization
As mentioned, we want to make sure every vector space has an orthonormal
basis, and we can extend any orthonormal set to an orthonormal basis, at least
in the case of finite-dimensional vector spaces. The idea is to start with an
arbitrary basis, which we know exists, and produce an orthonormal basis out of
it. The way to do this is the Gram-Schmidt process.
Theorem
(Gram-Schmidt process)
.
Let
V
be an inner product space and
e
1
, e
2
, ···
a linearly independent set. Then we can construct an orthonormal set
v
1
, v
2
, ··· with the property that
hv
1
, ··· , v
k
i = he
1
, ··· , e
k
i
for every k.
Note that we are not requiring the set to be finite. We are just requiring it
to be countable.
Proof.
We construct it iteratively, and prove this by induction on
k
. The base
case k = 0 is contentless.
Suppose we have already found v
1
, ··· , v
k
that satisfies the properties. We
define
u
k+1
= e
k+1
−
k
X
i=1
(v
i
, e
i+1
)v
i
.
We want to prove that this is orthogonal to all the other
v
i
’s for
i ≤ k
. We have
(v
j
, u
k+1
) = (v
j
, e
k+1
) −
k
X
i=1
(v
i
, e
k+1
)δ
ij
= (v
j
, e
k+1
) − (v
j
, e
k+1
) = 0.
So it is orthogonal.
We want to argue that u
k+1
is non-zero. Note that
hv
1
, ··· , v
k
, u
k+1
i = hv
1
, ··· , v
k
, e
k+1
i
since we can recover
e
k+1
from
v
1
, ··· , v
k
and
u
k+1
by construction. We also
know
hv
1
, ··· , v
k
, e
k+1
i = he
1
, ··· , e
k
, e
k+1
i
by assumption. We know
he
1
, ··· , e
k
, e
k+1
i
has dimension
k
+ 1 since the
e
i
are
linearly independent. So we must have
u
k+1
non-zero, or else
hv
1
, ··· , v
k
i
will
be a set of size
k
spanning a space of dimension
k
+ 1, which is clearly nonsense.
Therefore, we can define
v
k+1
=
u
k+1
ku
k+1
k
.
Then
v
1
, ··· , v
k+1
is orthonormal and
hv
1
, ··· , v
k+1
i
=
he
1
, ··· , e
k+1
i
as re-
quired.
Corollary.
If
V
is a finite-dimensional inner product space, then any orthonor-
mal set can be extended to an orthonormal basis.
Proof.
Let
v
1
, ··· , v
k
be an orthonormal set. Since this is linearly independent,
we can extend it to a basis (v
1
, ··· , v
k
, x
k+1
, ··· , x
n
).
We now apply the Gram-Schmidt process to this basis to get an orthonormal
basis of
V
, say (
u
1
, ··· , u
n
). Moreover, we can check that the process does not
modify our v
1
, ··· , v
k
, i.e. u
i
= v
i
for 1 ≤ i ≤ k. So done.
Definition
(Orthogonal internal direct sum)
.
Let
V
be an inner product space
and
V
1
, V
2
≤ V
. Then
V
is the orthogonal internal direct sum of
V
1
and
V
2
if it
is a direct sum and V
1
and V
2
are orthogonal. More precisely, we require
(i) V = V
1
+ V
2
(ii) V
1
∩ V
2
= 0
(iii) (v
1
, v
2
) = 0 for all v
1
∈ V
1
and v
2
∈ V
2
.
Note that condition (iii) implies (ii), but we write it for the sake of explicitness.
We write V = V
1
⊥ V
2
.
Definition
(Orthogonal complement)
.
If
W ≤ V
is a subspace of an inner
product space V , then the orthogonal complement of W in V is the subspace
W
⊥
= {v ∈ V : (v, w ) = 0, ∀w ∈ W }.
It is true that the orthogonal complement is a complement and orthogonal,
i.e. V is the orthogonal direct sum of W and W
⊥
.
Proposition.
Let
V
be a finite-dimensional inner product space, and
W ≤ V
.
Then
V = W ⊥ W
⊥
.
Proof.
There are three things to prove, and we know (iii) implies (ii). Also, (iii)
is obvious by definition of W
⊥
. So it remains to prove (i), i.e. V = W + W
⊥
.
Let w
1
, ··· , w
k
be an orthonormal basis for W , and pick v ∈ V . Now let
w =
k
X
i=1
(w
i
, v)w
i
.
Clearly, we have
w ∈ W
. So we need to show
v − w ∈ W
⊥
. For each
j
, we can
compute
(w
j
, v − w) = (w
j
, v) −
k
X
i=1
(w
i
, v)(w
j
, w
i
)
= (w
j
, v) −
k
X
i=1
(w
i
, v)δ
ij
= 0.
Hence for any λ
i
, we have
X
λ
j
w
j
, v − w
= 0.
So we have v − w ∈ W
⊥
. So done.
Definition
(Orthogonal external direct sum)
.
Let
V
1
, V
2
be inner product spaces.
The orthogonal external direct sum of
V
1
and
V
2
is the vector space
V
1
⊕V
2
with
the inner product defined by
(v
1
+ v
2
, w
1
+ w
2
) = (v
1
, w
1
) + (v
2
, w
2
),
with v
1
, w
1
∈ V
1
, v
2
, w
2
∈ V
2
.
Here we write v
1
+ v
2
∈ V
1
⊕ V
2
instead of (v
1
, v
2
) to avoid confusion.
This external direct sum is equivalent to the internal direct sum of
{
(
v
1
, 0
) :
v
1
∈ V
1
} and {(0, v
2
) : v
2
∈ V
2
}.
Proposition.
Let
V
be a finite-dimensional inner product space and
W ≤ V
.
Let (
e
1
, ··· , e
k
) be an orthonormal basis of
W
. Let
π
be the orthonormal
projection of
V
onto
W
, i.e.
π
:
V → W
is a function that satisfies
ker π
=
W
⊥
,
π|
W
= id. Then
(i) π is given by the formula
π(v) =
k
X
i=1
(e
i
, v)e
i
.
(ii) For all v ∈ V, w ∈ W , we have
kv − π(v)k ≤ kv − wk,
with equality if and only if
π
(
v
) =
w
. This says
π
(
v
) is the point on
W
that is closest to v.
W
⊥
w
v
π(v)
Proof.
(i) Let v ∈ V , and define
w =
X
i=1
(e
i
, v)e
i
.
We want to show this is
π
(
v
). We need to show
v − w ∈ W
⊥
. We can
compute
(e
j
, v − w) = (e
j
, v) −
k
X
i=1
(e
i
, v)(e
j
, e
i
) = 0.
So v − w is orthogonal to every basis vector in w, i.e. v − w ∈ W
⊥
.So
π(v) = π(w) + π(v − w) = w
as required.
(ii)
This is just Pythagoras’ theorem. Note that if
x
and
y
are orthogonal,
then
kx + yk
2
= (x + y, x + y)
= (x, x) + (x, y) + (y, x) + (y.y)
= kxk
2
+ kyk
2
.
We apply this to our projection. For any w ∈ W , we have
kv − wk
2
= kv − π(v)k
2
+ kπ(v) − wk
2
≥ kv − π(v)k
2
with equality if and only if kπ(v) − wk = 0, i.e. π(v) = w.