1Estimation
IB Statistics
1.1 Estimators
The goal of estimation is as follows: we are given iid
X
1
, ··· , X
n
, and we know
that their probability density/mass function is
f
X
(
x
;
θ
) for some unknown
θ
.
We know
f
X
but not
θ
. For example, we might know that they follow a Poisson
distribution, but we do not know what the mean is. The objective is to estimate
the value of θ.
Definition (Statistic). A statistic is an estimate of
θ
. It is a function
T
of the
data. If we write the data as x = (
x
1
, ··· , x
n
), then our estimate is written as
ˆ
θ = T (x). T (X) is an estimator of θ.
The distribution of T = T (X) is the sampling distribution of the statistic.
Note that we adopt the convention where capital X denotes a random variable
and x is an observed value. So
T
(X) is a random variable and
T
(x) is a particular
value we obtain after experiments.
Example. Let X
1
, ··· , X
n
be iid N(µ, 1). A possible estimator for µ is
T (X) =
1
n
X
X
i
.
Then for any particular observed sample x, our estimate is
T (x) =
1
n
X
x
i
.
What is the sampling distribution of
T
? Recall from IA Probability that in
general, if
X
i
∼ N
(
µ
i
, σ
2
i
), then
P
X
i
∼ N
(
P
µ
i
,
P
σ
2
i
), which is something we
can prove by considering moment-generating functions.
So we have
T
(X)
∼ N
(
µ,
1
/n
). Note that by the Central Limit Theorem,
even if
X
i
were not normal, we still have approximately
T
(X)
∼ N
(
µ,
1
/n
) for
large values of
n
, but here we get exactly the normal distribution even for small
values of n.
The estimator
1
n
P
X
i
we had above is a rather sensible estimator. Of course,
we can also have silly estimators such as
T
(X) =
X
1
, or even
T
(X) = 0
.
32
always.
One way to decide if an estimator is silly is to look at its bias.
Definition (Bias). Let
ˆ
θ
=
T
(X) be an estimator of
θ
. The bias of
ˆ
θ
is the
difference between its expected value and true value.
bias(
ˆ
θ) = E
θ
(
ˆ
θ) − θ.
Note that the subscript
θ
does not represent the random variable, but the thing
we want to estimate. This is inconsistent with the use for, say, the probability
mass function.
An estimator is unbiased if it has no bias, i.e. E
θ
(
ˆ
θ) = θ.
To find out
E
θ
(
T
), we can either find the distribution of
T
and find its
expected value, or evaluate
T
as a function of X directly, and find its expected
value.
Example. In the above example, E
µ
(T ) = µ. So T is unbiased for µ.