7Directional derivative
IA Differential Equations
7.4 Classification of stationary points
At a stationary point x
0
, we know that
∇f
(x
0
) = 0. So at a point near the
stationary point,
f(x) ≈ f(x
0
) +
1
2
δx · H · δx,
where H = ∇∇f(x
0
) is the Hessian matrix.
At a minimum, Every point near x
0
has
f
(x)
> f
(x
0
), i.e.
δ
x
·H ·δ
x
>
0 for
all δx. We say δx · H · δx is positive definite.
Similarly, at a maximum,
δ
x
· H · δ
x
<
0 for all
δ
x. We say
δ
x
· H · δ
x is
negative definite.
At a saddle,
δ
x
· H · δ
x is indefinite, i.e. it can be positive, negative or zero
depending on the direction.
This, so far, is not helpful, since we do not have an easy way to know what
sign
δ
x
· H · δ
x could be. Now note that
H
=
∇∇f
is symmetric (because
f
xy
=
f
yx
). So
H
can be diagonalized (cf. Vectors and Matrices). With respect
to these axes in which H is diagonal (principal axes), we have
δx · H · δx = (δx, δy, ··· , δz)
λ
1
λ
2
.
.
.
λ
n
δx
δy
.
.
.
δz
= λ
1
(δx)
2
+ λ
2
(δy)
2
+ ··· + λ
n
(δz)
2
where λ
1
, λ
2
, ···λ
n
are the eigenvalues of H.
So for
δ
x
· H · δ
x to be positive-definite, we need
λ
i
>
0 for all
i
. Similarly,
it is negative-definite iff
λ
i
<
0 for all
i
. If eigenvalues have mixed sign, then it
is a saddle point.
Finally, if there is at least one zero eigenvalue, then we need further analysis
to determine the nature of the stationary point.
Apart from finding eigenvalues, another way to determine the definiteness is
using the signature.
Definition (Signature of Hessian matrix). The signature of
H
is the pattern of
the signs of the subdeterminants:
f
xx
|{z}
|H
1
|
,
f
xx
f
xy
f
yx
f
yy
| {z }
|H
2
|
, ··· ,
f
xx
f
xy
··· f
xz
f
yx
f
yy
··· f
yz
.
.
.
.
.
.
.
.
.
.
.
.
f
zx
f
zy
··· f
zz
| {z }
|H
n
|=|H|
Proposition.
H
is positive definite if and only if the signature is +
,
+
, ··· ,
+.
H
is negative definite if and only if the signature is
−,
+
, ··· ,
(
−
1)
n
. Otherwise,
H is indefinite.