AI&BigData Lab 2016. Александр Баев: Transfer learning - зачем, как и где.
1 like3,407 views
The document discusses transfer learning in neural networks, highlighting its application in fields like computer vision and natural language processing. It outlines the process of training models on a source task and adapting them for a target task, emphasizing the benefits of leveraging existing knowledge when data is limited. Various strategies for transfer learning and examples of practical applications are provided, as well as resources for pretrained models.
AI&BigData Lab 2016. Александр Баев: Transfer learning - зачем, как и где.
- 2. ProblemsML problems Computer Vision Natural Language Processing Signal processing Predicting Decision making MODEL INPUTS INFERENCE DS
- 3. Conventional approaches Domain specific features INPUTS Model fitted to specific features set INFERENCE Assumes domain specific research, conducted by appropriate expert (feature engineering) Looks like hacks for practical solutions
- 4. Neural Networks Solution INPUTS INFERENCE Simple pre- and postprocessing: scaling, shifting, etc.
- 5. Is Neural Networks is simple solution? SVM - Kernel type - Smoothness RANDOM FOREST - Number of trees - Trees’ depth NEURAL NETWORKS
- 6. Is Neural Networks simple solution? • A few dozens of thousands of samples – is not bad dataset • A few hundreds of thousands of samples – is good dataset • Overfitting is problem even for datasets with several millions samples
- 7. Transfer learning Learning system Learning system Knowledge Source task Target task Storing knowledge gained while solving one problem and applying it to a different but related problem
- 8. What is knowledge in Neural Networks
- 9. Transfer learning for Neural Networks 1) Train on Source Task 2) Remove last layers 3) Add new last layers 4) Train net for Target Task
- 11. Weights transfer Source task: For example, train VGG-16 for ImageNet classification (1.2*10^6 samples, 1000 classes) Target task: Remove last layer with 1000 outputs Put your own last layer for N classes => same topology, weights initialized by weights from source task’s net
- 12. When such flow can be used Low amount of data Source and Target tasks are similar
- 13. Transfer learning types • New classifier (for example, SVM) • Layer by Layer pretrain (for example, early autoencoders) • Finetuning (with lower LR)
- 14. Examples. “Plankton” and AlexNet “Plankton” National Data Science Bowl kaggle.com Classes: 121 with hierarchy Samples: ~30 000 AlexNet: 8 layers inputs: 227x227
- 15. Examples. “Pankton” and Cifar10 “Plankton” National Data Science Bowl kaggle.com Classes: 121 with hierarchy Samples: ~30 000 AlexNet: 4 layers inputs: 32x32
- 16. Example. “Drivers” and GoogleNet “Drivers” State Farm Distracted Driver Detection kaggle.com Classes: 121 with hierarchy Samples: ~30 000 AlexNet: 4 layers inputs: 32x32
- 17. Model Zoo
- 18. Model Zoo CAFFE : ~ 30 models for different problems https://github.com/BVLC/caffe/wiki/Model-Zoo MatConvNet: ~10 models for different problems http://www.vlfeat.org/matconvnet/pretrained/ TensorFlow: https://github.com/tensorflow/models Torch: https://github.com/facebook/fb.resnet.torch https://github.com/szagoruyko/loadcaffe