Xgboost: A Scalable Tree Boosting System - Explained
Download as PPTX, PDF1 like2,670 views
XGBoost is a widely-used scalable tree boosting system that excels in classification, regression, and learning-to-rank tasks, often outperforming competitors. It employs decision tree boosting with optimizations like column subsampling and approximate split finding to enhance performance and reduce overfitting. Further resources and tutorials are available to aid in understanding and implementation.
Xgboost: A Scalable Tree Boosting System - Explained
- 1. XGBoost: A Scalable Tree Boosting System Simon Lia-Jonassen
- 2. Motivation Used by majority of winning solutions on Kaggle, 2nd most popular method after DNN. Also used by 10 best teams in KDDCup’15. Applies to classification, regression and learning-to-rank tasks. Usually outperforms alternatives in an out-of-the-box setting. Combines a good theoretical foundation and a highly efficient implementation. So, how does it work?
- 3. Decision Tree Boosting Number of trees Tree function, maps to a set of leaf weights Instance features
- 4. Regularized Learning Objective Prediction loss Complexity penalty Number of leaves L2 regularization on leaves weights
- 5. Regularized Learning Objective First order gradient of the loss function Second order gradient of the loss function By additive definition Where: However, for example:
- 6. Regularized Learning Objective By expansion: For each instance For each leaf For each instance in the leaf
- 7. Regularized Learning Objective Optimal leaf weight for a fixed structure: By substitution:
- 8. Gradient Tree Boosting Before we split Left split Right split Split penalty
- 10. Optimizations Shrinkage More trees Column subsampling Prevents over-fitting Approximate split finding Faster AUC convergence Sparsity-aware split finding Visit only non-missing values Cache-aware parallel column block access Fewer misses on large datasets Block compression and sharding Faster I/O for out-of-core computation
- 11. Optimizations Shrinkage More trees Column subsampling Prevents over-fitting Approximate split finding Faster AUC convergence Sparsity-aware split finding Visit only non-missing values Cache-aware parallel column block access Fewer misses on large datasets Block compression and sharding Faster I/O for out-of-core computation
- 12. Optimizations Shrinkage More trees Column subsampling Prevents over-fitting Approximate split finding Faster AUC convergence Sparsity-aware split finding Visit only non-missing values Cache-aware parallel column block access Fewer misses on large datasets Block compression and sharding Faster I/O for out-of-core computation
- 13. Optimizations Shrinkage More trees Column subsampling Prevents over-fitting Approximate split finding Faster AUC convergence Sparsity-aware split finding Visit only non-missing values Cache-aware parallel column block access Fewer misses on large datasets Block compression and sharding Faster I/O for out-of-core computation
- 14. Optimizations Shrinkage More trees Column subsampling Prevents over-fitting Approximate split finding Faster AUC convergence Sparsity-aware split finding Visit only non-missing values Cache-aware parallel column block access Fewer misses on large datasets Block compression and sharding Faster I/O for out-of-core computation
- 15. Optimizations
- 16. Further reading The paper: https://arxiv.org/pdf/1603.02754.pdf XGBoost tutorial: http://xgboost.readthedocs.io/en/latest/model.html A great deck of slides: https://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf A simple usage example: https://www.kaggle.com/kevalm/xgboost-implementation-on-iris-dataset-python DataCamp mini-course: https://campus.datacamp.com/courses/extreme-gradient-boosting-with-xgboost