Bayesian learning
2 likes1,168 views
Bayesian learning allows systems to classify new examples based on a model learned from prior examples using probabilities. It reasons by calculating the posterior probability of a hypothesis given new evidence using Bayes' rule. The maximum a posteriori (MAP) hypothesis that best explains the evidence is selected. Naive Bayes classifiers make a strong independence assumption between attributes. They classify new examples by calculating the posterior probability of each class and choosing the class with the highest value. Overfitting can occur if the learned model is too complex for the data. Model selection aims to avoid overfitting by evaluating models on separate training and test datasets.
Bayesian learning
- 2. A sample learning task: Classification • The system is given a set of instances to learn from • The system builds/selects a model/hypothesis based on the given instances • Based on the learned model the system can classify unseen instances
- 3. Reasoning with probabilities )( )|()( )|( DP hDPhP DhP Bayes Rule evidence likelihoodprior posterior
- 4. Selecting the best hypothesis )|()(maxarg hDPhPh Hh MAP )( )|()( maxarg DP hDPhP h Hh MAP P(D) can be dropped because it's the same for every h MAP = Maximum A Posteriori
- 5. Probability of Poliposis 008.0)( poliposisP 98.0)|( poliposisP 0078.0008.098.0)|()( poliposisPpoliposisP ?)|( poliposisP Given a medical test is positive what's the probability we actually have polisposis disease? prior likelihood posterior
- 6. Probability of Poliposis 992.0)( poliposisP 03.0)|( poliposisP 0298.0992.003.0)|()( poliposisPpoliposisP ?)|( poliposisP prior likelihood posterior Given a medical test is positive what's the probability we DON'T have polisposis disease?
- 7. Probability of Poliposis 0078.0)|()( poliposisPpoliposisP 0298.0)|()( poliposisPpoliposisP 21.0)|( poliposisP By normalizing we obtain the above probabilities. It's approximately 4 times more likely we DON'T have polisposis By comparing the posteriors we find the map hypothesis 79.0)|( poliposisP
- 8. Learning from examples Day Outlook Temp Humidity Wind Play Hacky Sack 1 Sunny Hot High Weak Yes 2 Sunny Hot High Strong No 3 Overcast Hot High Strong Yes 4 Rain Mild High Strong no 5 Rain Cool Normal Strong No 6 Rain Cool Normal Strong No
- 9. Naive Bayes classifier )|()(maxarg hDPhPh Hh MAP i kik Kk NB CxPCPC )|()(maxarg Called naive because it assumes all attributes are independent
- 10. Implicit model Day Outlook Temp Humidity Wind Class 1 Sunny = 1/2 Hot = 2/2 High = 2/2 Weak = 1/2 Yes= 2/6 3 Overcast= 1/2 Strong = 1/2 2 Sunny = 1/4 Hot = 1/4 High = 2/4 Strong = 4/4 No = 4/6 4 Rain = 3/4 Mild = 1/4 5 Cool = 2/4 Normal = 2/4 6
- 11. Classifying an example 333.06/2)( yesackPlayHackySP Day Outlook Temp Humidity Wind Play Hacky Sack? 7 Sunny Hot High Strong ? 666.06/4)( noackPlayHackySP 5.02/1)|( yesackPlayHackySsunnyOutlookP 25.04/1)|( noackPlayHackySsunnyOutlookP 12/2)|( yesackPlayHackyShotTempP 25.04/1)|( noackPlayHackyShotTempP 12/2)|( yesackPlayHackyShighHumidityP 5.04/2)|( noackPlayHackyShighHumidityP 5.02/1)|( yesackPlayHackySstrongWindP 14/4)|( noackPlayHackySstrongWindP prior likelihood
- 12. Result 08325.05.0115.0333.0)|()|()|()|()( yesstrongPyeshighPyeshotPyessunnyPyesP 8.0! yesackPlayHackyS 0208125.015.025.025.0666.0)|()|()|()|()( nostrongPnohighPnohotPnosunnyPnoP Calculating the MAP hypothesis for class 'yes' & 'no'
- 13. Least square error hypothesis Where the red lines represents the error
- 14. Choosing between two hypotheses red/yellow lines are the error for h1/h2
- 15. Maximum Likelihood Hypothesis )|(maxarg hDPh Hh ML m i i Hh ML hdPh 1 )|(maxarg Assuming all data points are independent Assuming uniform priors
- 16. Least square error hypothesis 2 2 ))(( 2 1 1 2 2 1 maxarg ii xhdm iHh ML eh using a least square error method has a profound theoretical basis m i ii Hh ML xhdh 1 2 ))((minarg assuming the noise is normally distributed
- 17. Squared error analysis x error h1 error h2 square e1 square e2 2 -1.5 2 2.25 4 4 4.5 6 20.25 36 6 -1.5 -2 2.25 4 8 1.5 -1 2.25 1 10 6.5 2 42.25 4 product 9745.39 2304 h2 is more likely then h1 since it has a smaller squared error product
- 18. Overfitting The model found by the learning algorithm is too complex
- 19. Model selection • Usually tested by splitting the data into training and testing examples • Do model selection based on different data sets • Prefer simple hypothesis over more complex ones (assign lower priors to complex hypothesis)
- 20. Minimum Description Length learning (MDL) • Encode both hypothesis and data using an optimal encoding will output the MAP hypothesis. )|()(minarg 21 hDLhLh CC Hh MDL MAPMDL hh Compression, Probability, Regularity are all closely related!
- 21. References • Tom M. Mitchell (1997) Machine Learning (pp 154-200)