1Estimation

IB Statistics

1.1 Estimators

The goal of estimation is as follows: we are given iid

X

1

, ··· , X

n

, and we know

that their probability density/mass function is

f

X

(

x

;

θ

) for some unknown

θ

.

We know

f

X

but not

θ

. For example, we might know that they follow a Poisson

distribution, but we do not know what the mean is. The objective is to estimate

the value of θ.

Definition (Statistic). A statistic is an estimate of

θ

. It is a function

T

of the

data. If we write the data as x = (

x

1

, ··· , x

n

), then our estimate is written as

ˆ

θ = T (x). T (X) is an estimator of θ.

The distribution of T = T (X) is the sampling distribution of the statistic.

Note that we adopt the convention where capital X denotes a random variable

and x is an observed value. So

T

(X) is a random variable and

T

(x) is a particular

value we obtain after experiments.

Example. Let X

1

, ··· , X

n

be iid N(µ, 1). A possible estimator for µ is

T (X) =

1

n

X

X

i

.

Then for any particular observed sample x, our estimate is

T (x) =

1

n

X

x

i

.

What is the sampling distribution of

T

? Recall from IA Probability that in

general, if

X

i

∼ N

(

µ

i

, σ

2

i

), then

P

X

i

∼ N

(

P

µ

i

,

P

σ

2

i

), which is something we

can prove by considering moment-generating functions.

So we have

T

(X)

∼ N

(

µ,

1

/n

). Note that by the Central Limit Theorem,

even if

X

i

were not normal, we still have approximately

T

(X)

∼ N

(

µ,

1

/n

) for

large values of

n

, but here we get exactly the normal distribution even for small

values of n.

The estimator

1

n

P

X

i

we had above is a rather sensible estimator. Of course,

we can also have silly estimators such as

T

(X) =

X

1

, or even

T

(X) = 0

.

32

always.

One way to decide if an estimator is silly is to look at its bias.

Definition (Bias). Let

ˆ

θ

=

T

(X) be an estimator of

θ

. The bias of

ˆ

θ

is the

difference between its expected value and true value.

bias(

ˆ

θ) = E

θ

(

ˆ

θ) − θ.

Note that the subscript

θ

does not represent the random variable, but the thing

we want to estimate. This is inconsistent with the use for, say, the probability

mass function.

An estimator is unbiased if it has no bias, i.e. E

θ

(

ˆ

θ) = θ.

To find out

E

θ

(

T

), we can either find the distribution of

T

and find its

expected value, or evaluate

T

as a function of X directly, and find its expected

value.

Example. In the above example, E

µ

(T ) = µ. So T is unbiased for µ.