![Breakthrough, [Hinton, et al., 2006]
Layer-Wise Pre-Training, unsupervised
Optimize likelihood of data, P(x)
• Reference: Deep Learning and Neural Networks, by Kevin Duh,
http://cl.naist.jp/~kevinduh/a/deep2014/](https://image.slidesharecdn.com/introdeeplearning-150711004128-lva1-app6892/85/Intro-deep-learning-20-320.jpg)
![Extracted Features
Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations[Lee et
al., 2009]](https://image.slidesharecdn.com/introdeeplearning-150711004128-lva1-app6892/85/Intro-deep-learning-25-320.jpg)
Intro deep learning
- 1. M. R.Avendi CPCC, UC Irvine Nov. 2014 Deep Learning, Trends, and Advances
- 2. Outline Introduction and Motivations Machine Learning and Challenges Neuroscience Experiments Neural Networks and Optimization Deep Networks and Advances Summary
- 3. Machine Learning Supervised: labeled data, eg. spam filtering Unsupervised: unlabeled data, eg. clustering
- 5. Images, Behind the Scene
- 7. Image Classification What are features!?
- 8. Feature Extraction Hard , time consuming, requires knowledge Human brain does feature extraction
- 9. Neuroscience Experiment, (1992) Auditory cortex learns to see! Roe, Anna W., et al. "Visual projections routed to the auditory pathway in ferrets: receptive fields of visual neurons in primary auditory cortex." The Journal of neuroscience 12.9 (1992): 3651-3664.
- 10. Seeing With Tongue Blind people can see using tongue http://www.wicab.com/en_us/press.html
- 11. One Learning Algorithm Hypothesis
- 12. We want: Automatic feature learning Training data Unlabeled: whatever, we have a lot! Labeled: small!
- 13. Mimicking Brain: Neural Networks Perceptron: one-layer NN Parameters : w, not known, training Activation function: f(x) =f(wi xi +w0)
- 14. Training One Layer Network Training data: input x, output y • Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/
- 15. Training: Gradient Descent • Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/
- 16. Two-Layer Network • Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/
- 17. Training: Backpropagation • Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/
- 18. Training: Backpropagation • Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/
- 19. Multi-Layer Network: Dark Ages • Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/
- 20. Breakthrough, [Hinton, et al., 2006] Layer-Wise Pre-Training, unsupervised Optimize likelihood of data, P(x) • Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/
- 21. Breakthrough, cnt. Fine-tune using labeled data, supervised • Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/
- 22. Deep Learning Approaches Deep Belief Networks RBM: learns data likelihood Stacked RBMs
- 23. Deep Learning Approaches Stacked Autoencoders Autoencoders: learns to reconstruct input data Easier to train • Reference: Deep Learning tutorial,Andrew Ng
- 24. Convolutional Networks • Reference: Deep Learning tutorial,Andrew Ng
- 25. Extracted Features Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations[Lee et al., 2009]
- 26. Deep Learning: advances Microsoft real-time speech translation https://www.youtube.com/watch?v=NhxCg2PA3ZI
- 27. Deep Learning: advances Google artificial brain learns to find cat and face NN, 1 billion connection, 16000 computers, browseYouTube for 3 days
- 28. Others Google+ Image Search, no-tag image search Handwriting recognition Android speech to text Medical Diagnosis
- 29. Summary • Reference: Deep Learning and Neural Networks, by Kevin Duh, http://cl.naist.jp/~kevinduh/a/deep2014/