Build an efficient Machine Learning model with LightGBM
0 likes2,165 views
Poo Kuan Hoong gives a presentation on building effective machine learning models with LightGBM. He begins with an introduction to decision trees and ensemble methods like gradient boosting. He explains that LightGBM is a gradient boosting framework that is faster and more accurate than other algorithms. It grows trees vertically rather than horizontally for increased speed and accuracy. Tips are provided for fine-tuning LightGBM like adjusting the number of leaves, learning rate, and using techniques like bagging and feature sub-sampling. A demo is then shown on a Kaggle dataset to predict safe drivers.
Build an efficient Machine Learning model with LightGBM
- 1. Poo Kuan Hoong Build an effective Machine Learning Model with LightGBM
- 2. Agenda • Introduction • Decision Tree • Ensemble Method • Gradient Boosting • Motivation for Gradient Boosting on Decision Trees • LightGBM • Demo
- 3. About Me Poo Kuan Hoong • Google Developer Expert (GDE) in Machine Learning • Founded and managing Malaysia R User Group & TensorFlow & Deep Learning Malaysia User Group
- 4. Malaysia R User Group https://www.facebook.com/groups/MalaysiaRUserGroup/
- 6. Introduction • Everyone is jumping into the hype of Deep Learning. • However, Deep Learning is not always the best model. • Deep Learning requires a lot of data, hyperparameters tuning and training time • Often, the best model is the simplest model.
- 8. Goal 1. Partition input space 2. Pure class distribution in each partition
- 9. Decision Trees: Guillotine cuts
- 10. Decision Trees: Guillotine cuts
- 11. Decision Trees: Guillotine cuts
- 12. Finding The Best Split
- 13. Finding The Best Split
- 14. Finding The Best Split
- 15. Finding the best split
- 16. Greedily Constructing A Decision Tree
- 17. Greedily Constructing A Decision Tree
- 18. Greedily Constructing A Decision Tree
- 19. Greedily Constructing A Decision Tree
- 20. Ensemble Methods 1. Weighted combination of weak learners 2. Prediction is based on committee votes 3. Boosting: 1. Train ensemble one weak learner at the time 2. Focus new learners on wrongly predicted examples
- 21. Gradient Boosting 1. Learn a regressor 2. Compute the error residual (Gradient in deep learning) 3. Then build a new model to predict that residual
- 22. Motivation for gradient boosting on Decision Trees Single decision tree can easily overfit the data
- 24. Gradient boosting on decision trees • Let’s define our objective functions
- 25. Gradient boosting on decision trees – regularization
- 26. Tricks from XGBoost • The tree is grown in breadth first fashion (as opposed to depth first like in the original C4.5 implementation). This provides a possibility of sorting and traversing data only once on each level • Furthermore, the sorted features can be cached – no need to sort that many times
- 27. LightGBM • LightGBM is a fast, distributed, high-performance gradient boosting framework based on decision tree algorithm, used for ranking, classification and many other machine learning tasks. • New library, developed by Microsoft, part of Distributed Machine Learning Toolkit. • Main idea: make the training faster First release: April, 24th 2017
- 28. Gradient Boosting Machine (GBM)
- 29. Why LightGBM? • Light GBM grows tree vertically while other algorithm grows trees horizontally meaning that Light GBM grows tree leaf- wise while other algorithm grows level- wise. • It will choose the leaf with max delta loss to grow. When growing the same leaf, Leaf-wise algorithm can reduce more loss than a level-wise algorithm
- 30. Features Speed • Light GBM is prefixed as ‘Light’ because of its high speed. Light GBM can handle the large size of data and takes lower memory to run Accuracy • LightGBM focuses on accuracy of results. Distributed/Parellel Computing • LGBM also supports GPU learning
- 31. Tips to fine tune LightGBM • Following set of practices can be used to improve your model efficiency. • num_leaves: This is the main parameter to control the complexity of the tree model. Ideally, the value of num_leaves should be less than or equal to 2^(max_depth). Value more than this will result in overfitting. • min_data_in_leaf: Setting it to a large value can avoid growing too deep a tree, but may cause under-fitting. In practice, setting it to hundreds or thousands is enough for a large dataset. • max_depth: You also can use max_depth to limit the tree depth explicitly.
- 32. Tips to fine tune LightGBM • For Faster Speed: • Use bagging by setting bagging_fraction and bagging_freq • Use feature sub-sampling by setting feature_fraction • Use small max_bin • Use save_binary to speed up data loading in future learning • Use parallel learning, refer to parallel learning guide.
- 33. Tips to fine tune LightGBM • For better accuracy: • Use large max_bin (may be slower) • Use small learning_rate with large num_iterations • Use large num_leaves (may cause over-fitting) • Use bigger training data • Try dart • Try to use categorical feature directly
- 34. Conclusion • LightGBM works well on multiple datasets and its accuracy is as good or even better than other boosting algorithms. • Based on its speed and accuracy, it is recommended to try LightGBM
- 35. To install LightGBM R Package • Build and install R-package with the following commands: git clone --recursive https://github.com/Microsoft/LightGBM cd LightGBM Rscript build_r.R https://github.com/Microsoft/LightGBM/tree/master/R-package
- 36. DEMO
- 37. Data • Porto Seguro’s Safe Driver Prediction • https://www.kaggle.com/c/porto-seguro-safe-driver-prediction