1Multivariate calculus
IB Variational Principles
1.1 Stationary points
The quest of minimization starts with finding stationary points.
Definition
(Stationary points)
.
Stationary points are points in
R
n
for which
∇f = 0, i.e.
∂f
∂x
1
=
∂f
∂x
2
= ··· =
∂f
∂x
n
= 0
All minima and maxima are stationary points, but knowing that a point is
stationary is not sufficient to determine which type it is. To know more about
the nature of a stationary point, we Taylor expand
f
about such a point, which
we assume is 0 for notational convenience.
f(x) = f(0) + x · ∇f +
1
2
X
i,j
x
i
x
j
∂
2
f
∂x
i
∂x
j
+ O(x
3
).
= f(0) +
1
2
X
i,j
x
i
x
j
∂
2
f
∂x
i
∂x
j
+ O(x
3
).
The second term is so important that we have a name for it:
Definition (Hessian matrix). The Hessian matrix is
H
ij
(x) =
∂
2
f
∂x
i
∂x
j
Using summation notation, we can write our result as
f(x) − f(0) =
1
2
x
i
H
ij
x
j
+ O(x
3
).
Since
H
is symmetric, it is diagonalizable. Thus after rotating our axes to a
suitable coordinate system, we have
H
0
ij
=
λ
1
0 ··· 0
0 λ
2
··· 0
.
.
.
.
.
.
.
.
.
.
.
.
0 0 ··· λ
n
,
where
λ
i
are the eigenvalues of
H
. Since
H
is real symmetric, these are all real.
In our new coordinate system, we have
f(x) − f(0) =
1
2
n
X
i=1
λ
i
(x
0
i
)
2
This is useful information. If all eigenvalues
λ
i
are positive, then
f
(
x
)
− f
(
0
)
must be positive (for small
x
). Hence our stationary point is a local minimum.
Similarly, if all eigenvalues are negative, then it is a local maximum.
If there are mixed signs, say
λ
1
>
0 and
λ
2
<
0, then
f
increases in the
x
1
direction and decreases in the
x
2
direction. In this case we say we have a saddle
point.
If some
λ
= 0, then we have a degenerate stationary point. To identify the
nature of this point, we must look at even higher derivatives.
In the special case where
n
= 2, we do not need to explicitly find the
eigenvalues. We know that
det H
is the product of the two eigenvalues. Hence if
det H
is negative, the eigenvalues have different signs, and we have a saddle. If
det H is positive, then the eigenvalues are of the same sign.
To determine if it is a maximum or minimum, we can look at the trace of
H
, which is the sum of eigenvalues. If
tr H
is positive, then we have a local
minimum. Otherwise, it is a local maximum.
Example. Let f(x, y) = x
3
+ y
3
− 3xy. Then
∇f = 3(x
2
− y, y
2
− x).
This is zero iff
x
2
=
y
and
y
2
=
x
. This is satisfied iff
y
4
=
y
. So either
y
= 0,
or y = 1. So there are two stationary points: (0, 0) and (1, 1).
The Hessian matrix is
H =
6x −3
−3 6y
,
and we have
det H = 9(4xy −1)
tr H = 6(x + y).
At (0
,
0),
det H <
0. So this is a saddle point. At (1
,
1),
det H >
0,
tr H >
0.
So this is a local minimum.