TreeNet Tree Ensembles & CART Decision Trees: A Winning Combination
Download as PPTX, PDF0 likes519 views
The document discusses the strengths and weaknesses of CART decision trees and Treenet stochastic gradient boosting, emphasizing their combined advantages for predictive modeling. A demonstration dataset of bank customers is used to illustrate how both methods can effectively identify individuals in bad standing by leveraging various financial predictors. Treenet enhances CART by improving accuracy while addressing its limitations like instability and data fragmentation.
TreeNet Tree Ensembles & CART Decision Trees: A Winning Combination
- 1. TreeNet Tree Ensembles and CART Decision Trees: A Winning Combination October 2012 Mikhail Golovnya Salford Systems CART® software is a trademark of California Statistical Software, Inc. and is licensed exclusively to Salford Systems. TreeNet® software is a trademark of Salford Systems
- 2. Course Outline • CART decision tree pros/cons • TreeNet stochastic gradient boosting: a promising way to overcome the shortcomings of a single tree • Introducing TreeNet, a powerful modern ensemble of boosted trees o Methodology o Reporting o Interpretability o Post-processing o Interaction detection • Advantages of using both CART and TreeNet o Contribution from CART o Contribution from TreeNet © Salford Systems 2012
- 3. Demonstration Dataset 108,376 bank customers (commercial and individual) with 6,564 in bad standing over the past two years Goal: identify customers in bad standing using the following predictors Revolving utilization of credit Age of the primary account holder Debt ratio of the primary account holder Monthly income Number of open credit lines Number of mortgages Number of dependents © Salford Systems 2012
- 4. CART Advantages 1. Relatively fast 2. All types of variables 1. Numeric, binary, categorical, missing values 3. Invariant under monotone transformations 1. Variable scales are irrelevant 2. Immunity to outliers 3. Most variables can be used “as is” 4. Resistance to many irrelevant variables 5. Few tunable parameters “off-the-shelf” procedure 6. Interpretable model representation © Salford Systems 2012
- 5. CART Disadvantages 1. Trade-off: accuracy vs. interpretability 2. Piecewise-constant model 1. Big errors near region boundaries 2. Impossible to detect fine differences within the segment 3. Instability => high variance 1. Small data change => big model change (especially for large trees) 4. Data fragmentation – splitting 5. High interaction order model, unreasonably complicated way to represent simple additive dependencies © Salford Systems 2012
- 6. TreeNet Tree Ensembles • Complements CART advantages, while dramatically increasing accuracy Tree 1 Tree 2 Tree 3 + + First tree grown 2nd tree grown on 3rd tree grown on on original residuals from residuals from target. first. Predictions model consisting Intentionally made to improve of first two trees “weak” model first tree © Salford Systems 2012
- 7. TreeNet Overcomes CART’s Shortcomings Piecewise-Constant CART TreeNet Model Big errors near region Fine predictions, nearly boundaries, coarse emulating smooth predictions continuous response surface Instability and Variance CART TreeNet Small data changes Stable models due to induce big model changes averaging of individual (especially for large trees) tree responses Data Fragmentation CART TreeNet Relatively few predictors Each tree works with the make it into the model entire data – many opportunities for variables to enter High Interaction Order CART TreeNet Model Always enforced Allows precise control © Salford Systems 2012 over the interactions
- 8. TreeNet and CART A Winning Combination © Salford Systems 2012