Covering algorithm
Download as PPTX, PDF0 likes663 views
The document discusses covering (rule-based) algorithms for generating classification rules from data. It provides an example of using a simple covering algorithm to iteratively generate rules that assign contact lens recommendations based on patient attributes. The algorithm works by selecting the test at each step that best separates the data (maximizes accuracy) until all instances are covered by rules or no further separation is possible.
Covering algorithm
- 2. Example: generating a rule • Possible rule set for class “a” • If x > 1.2 then class = a • If x > 1.2 and y > 2.6 then class = a • Possible rule set for class “b”: • If x ≤ 1.2 then class = b • If x > 1.2 and y ≤ 2.6 then class = b • Could add more rules, get “perfect” rule set
- 3. • Corresponding decision tree: (produces exactly the same predictions) • But: rule sets can be more perspicuous when decision trees suffer from replicated subtrees • Also: in multiclass situations, covering algorithm concentrates on one class at a time whereas decision tree learner takes all classes into account Rules vs Trees
- 4. • Generates a rule by adding tests that maximize rule’s accuracy • Similar to situation in decision trees: problem of selecting an attribute to split on • Each new test reduces rule’s coverage: Simple covering algorithm
- 5. • Convert decision tree into a rule set – Straightforward, but rule set overly complex – More effective conversions are not trivial • Instead, can generate rule set directly – for each class in turn find rule set that covers all instances in it (excluding instances not in the class) • Called a covering approach: – at each stage a rule is identified that “covers” some of the instances Covering Algorithms
- 6. Selecting a test • Goal: maximize accuracy – t total number of instances covered by rule – p positive examples of the class covered by rule – t – p number of errors made by rule Select test that maximizes the ratio p/t • We are finished when p/t = 1 or the set of instances can’t be split any further
- 7. Age Spectacle prescription Astigmatism Tear production rate Recommended lenses Young Myope No Reduced None Young Myope No Normal Soft Young Myope Yes Reduced None Young Myope Yes Normal Hard Young Hypermetrope No Reduced None Young Hypermetrope No Normal Soft Young Hypermetrope Yes Reduced None Young Hypermetrope Yes Normal Hard Pre-presbyopic Myope No Reduced None Pre-presbyopic Myope No Normal Soft Pre-presbyopic Myope Yes Reduced None Pre-presbyopic Myope Yes Normal Hard Pre-presbyopic Hypermetrope No Reduced None Pre-presbyopic Hypermetrope No Normal Soft Pre-presbyopic Hypermetrope Yes Reduced None Pre-presbyopic Hypermetrope Yes Normal None Presbyopic Myope No Reduced None Presbyopic Myope No Normal None Presbyopic Myope Yes Reduced None Presbyopic Myope Yes Normal Hard Presbyopic Hypermetrope No Reduced None Presbyopic Hypermetrope No Normal Soft Presbyopic Hypermetrope Yes Reduced None Presbyopic Hypermetrope Yes Normal None
- 8. Example: Contact lens data • Rule we seek: • Possible tests: Age = Young Age = Pre-presbyopic Age = Presbyopic Spectacle prescription = Myope Spectacle prescription = Hypermetrope Astigmatism = no Astigmatism = yes Tear production rate = Reduced Tear production rate = Normal If ? then recommendation = hard
- 9. Example: Contact lens data • Rule we seek: • Possible tests: Age = Young 2/8 Age = Pre-presbyopic 1/8 Age = Presbyopic 1/8 Spectacle prescription = Myope 3/12 Spectacle prescription = Hypermetrope 1/12 Astigmatism = no 0/12 Astigmatism = yes 4/12 Tear production rate = Reduced 0/12 Tear production rate = Normal 4/12 If ? then recommendation = hard
- 10. Modified rule and resulting data • Rule with best test added: • Instances covered by modified rule: Age Spectacle prescription Astigmatism Tear production rate Recommended lenses Young Myope Yes Reduced None Young Myope Yes Normal Hard Young Hypermetrope Yes Reduced None Young Hypermetrope Yes Normal hard Pre-presbyopic Myope Yes Reduced None Pre-presbyopic Myope Yes Normal Hard Pre-presbyopic Hypermetrope Yes Reduced None Pre-presbyopic Hypermetrope Yes Normal None Presbyopic Myope Yes Reduced None Presbyopic Myope Yes Normal Hard Presbyopic Hypermetrope Yes Reduced None Presbyopic Hypermetrope Yes Normal None If astigmatism = yes then recommendation = hard
- 11. Further refinement • Current state: • Possible tests: Age = Young Age = Pre-presbyopic Age = Presbyopic Spectacle prescription = Myope Spectacle prescription = Hypermetrope Tear production rate = Reduced Tear production rate = Normal If astigmatism = yes and ? then recommendation = hard
- 12. Further refinement • Current state: • Possible tests: Age = Young 2/4 Age = Pre-presbyopic 1/4 Age = Presbyopic 1/4 Spectacle prescription = Myope 3/6 Spectacle prescription = Hypermetrope 1/6 Tear production rate = Reduced 0/6 Tear production rate = Normal 4/6 If astigmatism = yes and ? then recommendation = hard
- 13. Modified rule and resulting data • Rule with best test added: • Instances covered by modified rule: Age Spectacle prescription Astigmatism Tear production rate Recommended lenses Young Myope Yes Normal Hard Young Hypermetrope Yes Normal hard Pre-presbyopic Myope Yes Normal Hard Pre-presbyopic Hypermetrope Yes Normal None Presbyopic Myope Yes Normal Hard Presbyopic Hypermetrope Yes Normal None If astigmatism = yes and tear production rate = normal then recommendation = hard
- 14. Further refinement • Current state: • Possible tests: • Tie between the first and the fourth test – We choose the one with greater coverage Age = Young 2/2 Age = Pre-presbyopic 1/2 Age = Presbyopic 1/2 Spectacle prescription = Myope 3/3 Spectacle prescription = Hypermetrope 1/3 If astigmatism = yes and tear production rate = normal and ? then recommendation = hard
- 15. The result • Final rule: • Second rule for recommending “hard lenses”: (built from instances not covered by first rule) If astigmatism = yes and tear production rate = normal and spectacle prescription = myope then recommendation = hard
- 16. Age Spectacle prescription Astigmatism Tear production rate Recommended lenses Young Myope No Reduced None Young Myope No Normal Soft Young Myope Yes Reduced None Young Myope Yes Normal Hard Young Hypermetrope No Reduced None Young Hypermetrope No Normal Soft Young Hypermetrope Yes Reduced None Young Hypermetrope Yes Normal Hard Pre-presbyopic Myope No Reduced None Pre-presbyopic Myope No Normal Soft Pre-presbyopic Myope Yes Reduced None Pre-presbyopic Myope Yes Normal Hard Pre-presbyopic Hypermetrope No Reduced None Pre-presbyopic Hypermetrope No Normal Soft Pre-presbyopic Hypermetrope Yes Reduced None Pre-presbyopic Hypermetrope Yes Normal None Presbyopic Myope No Reduced None Presbyopic Myope No Normal None Presbyopic Myope Yes Reduced None Presbyopic Myope Yes Normal Hard Presbyopic Hypermetrope No Reduced None Presbyopic Hypermetrope No Normal Soft Presbyopic Hypermetrope Yes Reduced None Presbyopic Hypermetrope Yes Normal None
- 17. Age Spectacle prescription Astigmatism Tear production rate Recommended lenses Young Myope No Reduced None Young Myope No Normal Soft Young Myope Yes Reduced None Young Myope Yes Normal Hard Young Hypermetrope No Reduced None Young Hypermetrope No Normal Soft Young Hypermetrope Yes Reduced None Young Hypermetrope Yes Normal Hard Pre-presbyopic Myope No Reduced None Pre-presbyopic Myope No Normal Soft Pre-presbyopic Myope Yes Reduced None Pre-presbyopic Myope Yes Normal Hard Pre-presbyopic Hypermetrope No Reduced None Pre-presbyopic Hypermetrope No Normal Soft Pre-presbyopic Hypermetrope Yes Reduced None Pre-presbyopic Hypermetrope Yes Normal None Presbyopic Myope No Reduced None Presbyopic Myope No Normal None Presbyopic Myope Yes Reduced None Presbyopic Myope Yes Normal Hard Presbyopic Hypermetrope No Reduced None Presbyopic Hypermetrope No Normal Soft Presbyopic Hypermetrope Yes Reduced None Presbyopic Hypermetrope Yes Normal None
- 18. Age Spectacle prescription Astigmatism Tear production rate Recommended lenses Young Myope No Reduced None Young Myope No Normal Soft Young Myope Yes Reduced None Young Hypermetrope No Reduced None Young Hypermetrope No Normal Soft Young Hypermetrope Yes Reduced None Young Hypermetrope Yes Normal Hard Pre-presbyopic Myope No Reduced None Pre-presbyopic Myope No Normal Soft Pre-presbyopic Myope Yes Reduced None Pre-presbyopic Hypermetrope No Reduced None Pre-presbyopic Hypermetrope No Normal Soft Pre-presbyopic Hypermetrope Yes Reduced None Pre-presbyopic Hypermetrope Yes Normal None Presbyopic Myope No Reduced None Presbyopic Myope No Normal None Presbyopic Myope Yes Reduced None Presbyopic Hypermetrope No Reduced None Presbyopic Hypermetrope No Normal Soft Presbyopic Hypermetrope Yes Reduced None Presbyopic Hypermetrope Yes Normal None
- 19. Example: Contact lens data • Rule we seek: • Possible tests: Age = Young Age = Pre-presbyopic Age = Presbyopic Spectacle prescription = Myope Spectacle prescription = Hypermetrope Astigmatism = no Astigmatism = yes Tear production rate = Reduced Tear production rate = Normal If ? then recommendation = hard
- 20. Example: Contact lens data • Rule we seek: • Possible tests: Age = Young 1/7 Age = Pre-presbyopic 0/7 Age = Presbyopic 0/7 Spectacle prescription = Myope 0/9 Spectacle prescription = Hypermetrope 1/12 Astigmatism = no 0/12 Astigmatism = yes 1/9 Tear production rate = Reduced 0/12 Tear production rate = Normal 1/9 If ? then recommendation = hard
- 21. Modified rule and resulting data • Rule with best test added: • Instances covered by modified rule: If age = Young then recommendation = hard Age Spectacle prescription Astigmatism Tear production rate Recommended lenses Young Myope No Reduced None Young Myope No Normal Soft Young Myope Yes Reduced None Young Hypermetrope No Reduced None Young Hypermetrope No Normal Soft Young Hypermetrope Yes Reduced None Young Hypermetrope Yes Normal Hard
- 22. Further refinement • Current state: • Possible tests: If age = Young and ? then recommendation = hard Spectacle prescription = Myope Spectacle prescription = Hypermetrope Astigmatism = no Astigmatism = yes Tear production rate = Reduced Tear production rate = Normal
- 23. Further refinement • Current state: • Possible tests: If age = Young and ? then recommendation = hard Spectacle prescription = Myope 0/3 Spectacle prescription = Hypermetrope 1/4 Astigmatism = no 0/4 Astigmatism = yes 1/3 Tear production rate = Reduced 0/4 Tear production rate = Normal 1/3
- 24. Modified rule and resulting data • Rule with best test added: • Instances covered by modified rule: If age = Young and Astigmatism = yes then recommendation = hard Age Spectacle prescription Astigmatism Tear production rate Recommended lenses Young Myope Yes Reduced None Young Hypermetrope Yes Reduced None Young Hypermetrope Yes Normal Hard
- 25. Further refinement • Current state: • Possible tests: Spectacle prescription = Myope Spectacle prescription = Hypermetrope Tear production rate = Reduced Tear production rate = Normal If age = Young and Astigmatism = yes and ? then recommendation = hard
- 26. Further refinement • Current state: • Possible tests: Spectacle prescription = Myope 0/1 Spectacle prescription = Hypermetrope 1/2 Tear production rate = Reduced 0/2 Tear production rate = Normal 1/1 If age = Young and Astigmatism = yes and ? then recommendation = hard
- 27. Final Results If age = Young and astigmatism = yes and tear production rate=normal then recommendation = hard Age Spectacle prescription Astigmatism Tear production rate Recommended lenses Young Myope Yes Normal Hard Young Hypermetrope Yes Normal Hard If astigmatism = yes and tear production rate = normal and spectacle prescription = myope then recommendation = hard Age Spectacle prescription Astigmatism Tear production rate Recommended lenses Young Myope Yes Normal Hard Pre-presbyopic Myope Yes Normal Hard Presbyopic Myope Yes Normal Hard
- 28. Pseudo-code for PRISM For each class C Initialize E to the instance set While E contains instances in class C Create a rule R with an empty left-hand side that predicts class C Until R is perfect (or there are no more attributes to use) do For each attribute A not mentioned in R, and each value v, Consider adding the condition A = v to the left-hand side of R Select A and v to maximize the accuracy p/t (break ties by choosing the condition with the largest p) Add A = v to R Remove the instances covered by R from E
- 29. • PRISM with outer loop removed generates a decision list for one class – Subsequent rules are designed for rules that are not covered by previous rules – Order doesn’t matter because all rules predict the same class • Outer loop considers all classes separately – No order dependence implied
- 30. Separate and conquer • Methods like PRISM (for dealing with one class) are separate-and-conquer algorithms: – First, a rule is identified – Then, all instances covered by the rule are separated out – Finally, the remaining instances are “conquered” • Difference to divide-and-conquer methods: – Subset covered by rule doesn’t need to be explored any further