6More distributions
IA Probability
6.5 Multivariate normal
Let X
1
, ··· , X
n
be iid N(0, 1). Then their joint density is
g(x
1
, ··· , x
n
) =
n
Y
i=1
ϕ(x
i
)
=
n
Y
1
1
√
2π
e
−
1
2
x
2
i
=
1
(2π)
n/2
e
−
1
2
P
n
1
x
2
i
=
1
(2π)
n/2
e
−
1
2
x
T
x
,
where x = (x
1
, ··· , x
n
)
T
.
This result works if
X
1
, ··· , X
n
are iid
N
(0
,
1). Suppose we are interested in
Z = µ + AX,
where
A
is an invertible
n × n
matrix. We can think of this as
n
measurements
Z that are affected by underlying standard-normal factors X. Then
X = A
−1
(Z − µ)
and
|J| = |det(A
−1
)| =
1
det A
So
f(z
1
, ··· , z
n
) =
1
(2π)
n/2
1
det A
exp
−
1
2
(A
−1
(z − µ))
T
(A
−1
(z − µ))
=
1
(2π)
n/2
det A
exp
−
1
2
(z − µ)
T
Σ
−1
(z − µ)
=
1
(2π)
n/2
√
det Σ
exp
−
1
2
(z − µ)
T
Σ
−1
(z − µ)
.
where Σ = AA
T
and Σ
−1
= (A
−1
)
T
A
−1
. We say
Z =
Z
1
.
.
.
Z
n
∼ MV N(µ, Σ) or N(µ, Σ).
This is the multivariate normal.
What is this matrix Σ? Recall that
cov
(
Z
i
, Z
j
) =
E
[(
Z
i
− µ
i
)(
Z
j
− µ
j
)]. It
turns out this covariance is the i, jth entry of Σ, since
E[(Z − µ)(Z − µ)
T
] = E[AX(AX)
T
]
= E(AXX
T
A
T
) = AE[XX
T
]A
T
= AIA
T
= AA
T
= Σ
So we also call Σ the covariance matrix.
In the special case where n = 1, this is a normal distribution and Σ = σ
2
.
Now suppose
Z
1
, ··· , Z
n
have covariances 0. Then Σ =
diag
(
σ
2
1
, ··· , σ
2
n
).
Then
f(z
1
, ··· , z
n
) =
n
Y
1
1
√
2πσ
i
e
−
1
2σ
2
i
(z
i
−µ
i
)
2
.
So Z
1
, ··· , Z
n
are independent, with Z
i
∼ N(µ
i
, σ
2
i
).
Here we proved that if
cov
= 0, then the variables are independent. However,
this is only true when
Z
i
are multivariate normal. It is generally not true for
arbitrary distributions.
For these random variables that involve vectors, we will need to modify our
definition of moment generating functions. We define it to be
m(θ) = E[e
θ
T
X
] = E[e
θ
1
X
1
+···+θ
n
X
n
].
Bivariate normal
This is the special case of the multivariate normal when
n
= 2. Since there
aren’t too many terms, we can actually write them out.
The bivariate normal has
Σ =
σ
2
1
ρσ
1
σ
2
ρσ
1
σ
2
σ
2
2
.
Then
corr(X
1
, X
2
) =
cov(X
1
, X
2
)
p
var(X
1
) var(X
2
)
=
ρσ
1
σ
2
σ
1
σ
2
= ρ.
And
Σ
−1
=
1
1 − ρ
2
σ
−2
1
−ρσ
−1
1
σ
−1
2
−ρσ
−1
1
σ
−1
2
σ
−2
2
The joint pdf of the bivariate normal with zero mean is
f(x
1
, x
2
) =
1
2πσ
1
σ
2
p
1 − ρ
2
exp
−
1
2(1 − ρ
2
)
x
2
1
σ
2
1
−
2ρx
1
x
2
σ
1
σ
2
+
x
2
2
σ
2
2
If the mean is non-zero, replace x
i
with x
i
− µ
i
.
The joint mgf of the bivariate normal is
m(θ
1
, θ
2
) = e
θ
1
µ
1
+θ
2
µ
2
+
1
2
(θ
2
1
σ
2
1
+2θ
1
θ
2
ρσ
1
σ
2
+θ
2
2
σ
2
2
)
.
Nice and elegant.