Ml mle_bayes

Machine Learning
Maximum Likelihood Estimation
and Bayesian Parameter Estimation
(Parametric Learning)

Phong VO
vdphong@fit.hcmus.edu.vn

September 11, 2010

– Typeset by FoilTEX –

Introduction

• From previous lecture, designing classiﬁer assumes knowledge of p(x|ωi)
and P (ωi) for each class, i.e. For Gaussian densities, we need to know
µi, Σi for i = 1, . . . , c

• Unfortunately, this information is not available directly.

• Given training samples with true class label for each of sample, we have
a learning problem.

• If the form of the densities is known, i.e. the number of parameters and
general knowledge about the problem, a parameter estimation problem
results.

– Typeset by FoilTEX – 1

Example 1. Assume that p(x|ωi) is a normal density with mean µi and
covariance matrix Σi, although we do not know exact values of these
quantities. This knowledge simpliﬁes the problem from one of estimating
an unknow function p(x|ωx) to one of estimating the parameters µi and
Σi.


Approaches to Parameter Estimation

• In maximum likelihood estimation, we assume the parameters are ﬁxed,
but unknown. The MLE approach seeks the ”‘best”’ parameter estimate
in the sense that ”‘best”’ means the set of parameters that maximize
the probability of obtaining the training set.

• Bayesian estimation models the parameters to be estimated as random
variables with some (assumed) known priori distribution. The training
set are ”‘observation”’, which allow conversion of the a priori information
into an a posteriori density. The Bayesian approach uses the training set
to update the training set-conditioned density function of the unknown
parameters.


Maximum Likelihood Estimation

• MLE nearly always have good convergence properties as the number of
training samples increases.

• It is often simpler than alternate methods, such as Bayesian techniques.


Formulation

c
• Assume D = j=1 Dj , with the samples in Dj having been drawn
independently according to the probability law p(x|ωj ).

• Assume that p(x|ωj ) has a known parametric form, and is determined
uniquely by the value of a parameter vector θ j , i.e. p(x|ωi) ∼ N (µj , Σj )
where θ j = {µj , Σj }.

• The dependence of p(x|ωj ) on θ j is expressed as p(x|ωj , θ j ).

• Our problem: use the training samples to obtain good estimates for the
unknown parameter vectors θ 1, . . . , θ c.


• Assume more that samples in Di give no information about θ j if i = j.
In other word, parameters are functionally independent.

• The problem of classiﬁcation is turned into the problems of parameter
estimation: Use a set D of training samples draw independently from the
probability density p(x|θ) to estimate the unknown parameter vector θ.

• Suppose that D = {x1, . . . , xn}. Since the samples were drawn
independently, we have

n
p(D|θ) = p(xk |θ)
k=1

p(D|θ) is called the likelihood of θ with respect to the set of samples.


ˆ
• The maximum likelihood estimate of θ is the value θ that maximizes
p(D|θ)


• Take the logarithm on both sides (just for analytical purpose), we deﬁne
l(θ) as the log-likelihood function

l(θ) = ln p(D|θ)

ˆ
• Since the logarithm is monotonically increasing, the θ that maximizes
the log-likelihood also maximizes the likelihood,

n
ˆ
θ = arg max θ l(θ) = arg max θ ln p(xk |θ)
k=1

ˆ
• θ can be found by taking derivatives of log-likelihood function


n

θl = θ ln p(xk |θ)
k=1

where

 
∂
∂θ1
≡ .
. ,
 
θ
∂
∂θp

and then solve the equation

θl = 0.


• A solution could be a global maximum, a local maximum or minimum.
We have to check each of them individually.

• NOTE: A related class of estimators - maximum a posteriori or MAP
estimators - ﬁnd the value of θ that maximizes l(θ)p(θ). Thus a ML
estimator is a MAP estimator for the uniform or ”‘ﬂat”’ prior.


MLE: The Gaussian Case for Unknown µ

• In this case, only the mean is unknown. Under this condition, we consider
a sample point xk and ﬁnd

1 1
ln p(xk |µ) = − ln (2π)d|Σ| − (xk − µ)tΣ−1(xk − µ)
2 2
and

θ ln p(xk |µ) = Σ−1(xk − µ).

• The maximum likelihood estimate for µ must satisfy


n
Σ−1(xk − µ) = 0,
ˆ
k=1

• Solve above equation, we obtain

n
1
ˆ
µ= xk
n
k=1

• Interpretation: The maximum likelihood estimate for theu unknown
population mean is just the arithmetic average of the training samples
- the sample mean. Think of the n samples as a cloud of points, the
sample mean is the centroid of the cloud.


MLE: The Gaussian Case for Unknown µ and Σ

• Consider the univariate normal case, θ = {θ1, θ2} = {µ, σ 2}. The
log-likelihood of a single point is

1 1
ln p(xk |θ) = − ln 2πθ2 − (xk − θ1)2
2 2θ2

and its derivative is

1
θ2 (xk − θ1 )
θl = θ ln p(xk |θ) = 1 (xk −θ1 )2 .
− 2θ2 + 2θ2
2


• Let θl = 0 and we obtain

n
1 ˆ
(xk − θ1) = 0
ˆ
θ2
k=1

and

n n ˆ
1 (xk − θ1)2
− + =0
ˆ
θ2 ˆ2
θ
k=1 k=1 2
.

ˆ ˆ
• Substitute µ = θ1 and σ = θ2 we obtain

n
1
µ=
ˆ xk
n
k=1


and

n
2 1
σ =
ˆ (xk − µ)2
ˆ
n
k=1
.

Exercise 1. Estimate µ and Σ for the case of multivariate Gaussian.


Bayesian Parameter Estimation

• Bayes’ formula allows us to compute the posterior probabilities P (ωi|x)
from the prior probabilities P (ωi) and the class-conditional densities
p(x|ωi).

• How can we proceed those quantities?
– Prior probabilities: from knowledge of the functional forms for unknown
densities and ranges for the values of unknown parameters
– class-conditional densities: from training samples.


• Given training samples as D, Bayes’s formula then becomes

p(x|ωi, D)P (ωi|D)
P (ωi|x, D) = c
j=1 p(x|ωj , D)P (ωj |D)

• Assume that the a priori probabilities are known, P (ωi) = P (ωi|D) and
the samples in Di have no inﬂuence on p(x|ωj , D) if i = j,

p(x|ωi, Di)P (ωi)
P (ωi|x, D) = c
j=1 p(x|ωj , Dj )P (ωj )

• We have c separate problems of the following form: use a set D of samples
drawn independently according to the ﬁxed but unknown probability
distribution p(x) to determine p(x|D). Our supervised learning problem
is turned into an unsupervised density estimation problem.


The Parameter Distribution

• Although the desired probability density p(x) is unknown, we assume
that it has a known parametric form.

• The unknown factor is the value of a parameter vector θ. As long as θ
is known, the function p(x|θ) is known.

• Information that we have about θ prior to observing the samples is
assumed to be contained in a known prior density p(θ).

• Observation of the samples converts this to a posterior density p(θ|D),
which is expected to be sharply peaked about the true value of θ.


BPE: Gaussian Case

• Calculate p(θ|D) and p(x|D) for the case where p(x|µ) ∼ N (µ, Σ)

• Consider the univariate case where µ is unknown

p(x|µ) ∼ N (µ, σ 2)

• We assume that the prior density p(µ) has a known distribution

2
p(µ) ∼ N (µ0, σ0 )
2
Interpretation: µ0 represents our best a priori guess for µ, and σ0
measures our uncertainty about this guess.


• Once µ is ”‘guessed”’, it determines the density for x. Letting D =
{x1, . . . , xn}, Bayes’ formula gives us

n
p(D|µ)p(µ)
p(µ|D) = ∝ p(xk |µ)p(µ)
p(D|µ)p(µ)dµ
k=1

where it is easy to see the aﬀection of training samples to the estimation
of the true µ.

• Since p(xk |µ) ∼ N (µ, σ 2) and p(µ) ∼ N (µ0, σ0 ), we have
2


p(xk |µ) p(µ)
n 2 2
1 1 xk − µ 1 1 x k − µ0
p(µ|D) ∝ √ exp √ exp
σ 2π 2 σ σ0 2π 2 σ0
k=1
(4)
n 2 2
1 µ − xk µ − µ0
∝ exp − + (5)
2 σ σ0
k=1
n
1 n 1 1 µ0
∝ exp − + 2 µ2 − 2 xk + 2 µ (6)
2 σ 2 σ0 σ2 σ0
k=1

• p(µ|D) is again a normal density and is said to be a reproducing density
and p(µ) is conjugate prior.


2
• If we write p(µ|D) ∼ N (µn, σn), then

2
1 1 µ − µn
p(µ|D) = √ exp −
σ 2π 2 σn

• Equating coeﬃcients show us

2
nσ0 σ2
µn = 2 xn + 2 µ0
nσ0 + σ 2 nσ0 + σ 2
1 n
where xn = n k=1 xk and

2 σ0 σ 2
2
σn = 2
nσ0 + σ 2


Interpretation: these equations show how the prior information is
combined with the empirical information in the samples to obtain the a
posteriori density p(µ|D).


Interpretation

• µn represents our best guess for µ after observing n samples

2
• σn measures our uncertainty about this guess,

2 σ2
limn→∞ σn = ,
n
each additional observation decreases our uncertainty about the true
value of µ.

• As n increases, p(µ|D) approaches a Dirac delta function.

• This behavior is known as Bayesian learning.


• µn is a positive combination of xn and µ0, xn ≤ µn ≤ µ0

lim
 n→∞ µn = xn if σ = 0
µn = µ0 if σ0 = 0

xn if σ0 σ


• The dogmatism:
prior knowledge σ 2
∼ 2
empirical data σ0

• If the dogmatism in not inﬁnite, after enough samples are taken the exact
2
values assumed for µ0 and σ0 will be unimportant, µn will converge to
the sample mean.


Compute the class-conditional density

• Having obtained a posteriori density for the mean, p(µ|D), we now
compute the ”‘class-contitional”’ density for p(x|D)

p(x|D) = p(x|µ)p(µ|D)dµ (7)

1 1 x−µ 1 1 x − µn
= √ exp − √ exp −
σ 2π 2 σ σn 2π 2 σn
(8)
1 1 (x − µn)2
= exp − f (σ, σn), (9)
2πσσn 2 σ 2 + σn
2


where

2
1 σ 2 + σn 2
σn x + σ 2 µ n
2
f (σ, σn) = exp − 2σ 2
µ− dµ
2 σ n σ 2 + σn2

• Hence p(x|D) is normally distributed with mean µn and variance σ 2 + σn
2

p(x|D) ∼ N (µn, σ 2 + σn).
2

• The density p(x|D) is the desired class-conditional density p(x|ωj , Dj ).

Exercise 2. Use Bayesian estimation to calculate the a posteriori density
p(θ|D) and the desired probability density p(x|D) for the multivariate case
where p(x|µ) ∼ N (µ, Σ)


BPE: General Theory

The basic assumptions for the applicability of Bayesian estimation are
summarized as follows:

1. The form of the density p(x|θ) is assumed to be known, but the value
of the parameter vector θ is not known exactly.

2. Our initial knowledge about θ is assumed to be contained in a known a
priori density p(θ).

3. The rest of our knowledge about θ is contained in a set D of n samples
x1, . . . , xn drawn independently according to the unknown probability
density p(x).


The basic problem is to compute the posterior density p(θ|D)

p(x|D) = p(x|θ)p(θ|D)dθ.

By Bayes’ formula we have

p(D|θ)p(θ)
p(θ|D) =
p(D|θ)p(θ)dθ

and by the independence assumption

n
p(D|θ) = p(xk |θ).
k=1


Frequentists Perspective

• Probability refers to limiting relative frequencies. Probabilities are
objective properties of the real world.

• Parameters are fixed, unknown constants. Because they are not
fluctuating, no useful probability statements can be made about
parameters.

• Statistical procedures should be designed to have well-defined long run
frequency properties


Bayesian Perspective

• Probability describes degrees of belief, not limiting frequency.

• We can make probability statements about parameters, even though they
are ﬁxed constants.

• We make inference about a parameters θ by producing a probability
distribution for θ.


Frequentists VS. Bayesians

• Bayesian inference is a controversial approach because it inherently
embrace a subjective notion of probability.

•


Ml mle_bayes

More Related Content

What's hot (18)

Similar to Ml mle_bayes (20)

Ml mle_bayes