3Discrete random variables
IA Probability
3.3 Weak law of large numbers
Theorem (Weak law of large numbers). Let
X
1
, X
2
, ···
be iid random variables,
with mean µ and var σ
2
.
Let S
n
=
P
n
i=1
X
i
.
Then for all ε > 0,
P
S
n
n
− µ
≥ ε
→ 0
as n → ∞.
We say,
S
n
n
tends to µ (in probability), or
S
n
n
→
p
µ.
Proof. By Chebyshev,
P
S
n
n
− µ
≥ ε
≤
E
S
n
n
− µ
2
ε
2
=
1
n
2
E(S
n
− nµ)
2
ε
2
=
1
n
2
ε
2
var(S
n
)
=
n
n
2
ε
2
var(X
1
)
=
σ
2
nε
2
→ 0
Note that we cannot relax the “independent” condition. For example, if
X
1
=
X
2
=
X
3
=
···
= 1 or 0, each with probability 1
/
2. Then
S
n
/n →
1
/
2
since it is either 1 or 0.
Example. Suppose we toss a coin with probability p of heads. Then
S
n
n
=
number of heads
number of tosses
.
Since E[X
i
] = p, then the weak law of large number tells us that
S
n
n
→
p
p.
This means that as we toss more and more coins, the proportion of heads will
tend towards p.
Since we called the above the weak law, we also have the strong law, which
is a stronger statement.
Theorem (Strong law of large numbers).
P
S
n
n
→ µ as n → ∞
= 1.
We say
S
n
n
→
as
µ,
where “as” means “almost surely”.
It can be shown that the weak law follows from the strong law, but not the
other way round. The proof is left for Part II because it is too hard.