Introduction to Neural Network
9 likes1,218 views
The document summarizes a presentation on building artificial neural networks. It discusses an overview of machine learning algorithms that will be covered in upcoming sessions, including supervised and unsupervised learning methods as well as deep learning. It then provides details on feedforward neural networks, including their structure, how data is fed through the network, and how weights are learned through backpropagation and gradient descent. Applications discussed include voice recognition, object recognition, conversation bots, auto-driving cars, and gaming.
![•Gradient:
E[w]=[E/w0,… E/wn]
•(w1,w2)
•(w1+w1,w2 +w2)
•w=- E[w]
•wi=- E/wi
Learning: Gradient Descent
•Chain rule:](https://image.slidesharecdn.com/neuralnetworks-170409031214/85/Introduction-to-Neural-Network-20-320.jpg)
Introduction to Neural Network
- 1. Yan Xu Houston Machine Learning Meetup April 8, 2017 Building an artificial neural network
- 2. Roadmap: Method • Tour of machine learning algorithms (1 session) • Feature engineering (1 session) – Feature selection - Yan • Supervised learning (4 sessions) – Regression models -Yan – SVM and kernel SVM - Yan – Tree-based models - Dario – Bayesian method - Xiaoyang – Ensemble models - Yan • Unsupervised learning (3 sessions) – K-means clustering – DBSCAN - Cheng – Mean shift – Agglomerative clustering – Kunal – Spectral clustering – Yan – Dimension reduction for data visualization - Yan • Deep learning (4 sessions) _ Neural network - Yan – Convolutional neural network – Hengyang Lu – Recurrent neural networks – Train deep nets with open-source tools Slides posted on: http://www.slideshare.net/xuyangela
- 3. Outline • Introduction • Feed forward neural network • Learning process - by Geoffrey Hinton • Overfitting issue • Voice recognition • Bonus – Machine learning in Rock Facies Classification by Cheng Zhan
- 8. What are Neural Networks? • Models of the brain and nervous system • Highly parallel – Process information much more like the brain than a serial computer • Simple principles Complex behaviours
- 9. Biological Neural Nets • Pigeons as art experts (Watanabe et al. 1995) – Experiment: • Pigeon in Skinner box • Present paintings of two different artists (e.g. Chagall / Van Gogh) • Reward for pecking when presented a particular artist (e.g. Van Gogh)
- 11. • Pigeons were able to discriminate between Van Gogh and Chagall with 95% accuracy (when presented with pictures they had been trained on) • Discrimination still 85% successful for previously unseen paintings of the artists • Pigeons do not simply memorise the pictures • They can extract and recognise patterns (the ‘style’) • They generalise from the already seen to make predictions • This is what neural networks (biological and artificial) are good at (unlike conventional computer)
- 12. • ANNs incorporate the two fundamental components of biological neural nets: 1. Neurones (nodes) 2. Synapses (weights) Artificial Vs. Biological Neural Nets
- 13. • Neurone vs. Node
- 14. • Structure of a node: • Squashing function limits node output:
- 15. • Synapse vs. weight
- 16. Feed-forward nets • Information flow is unidirectional • Data is presented to Input layer • Passed on to Hidden Layer • Passed on to Output layer • Information is distributed • Information processing is parallel Internal representation (interpretation) of data
- 17. Feeding data through the net: (1 0.25) + (0.5 (-1.5)) = 0.25 + (-0.75) = - 0.5 0.3775 1 1 5.0 e Squashing:
- 18. • Data is presented to the network in the form of activations in the input layer • Examples – Pixel intensity (for pictures) – Molecule concentrations (for artificial nose) – Share prices (for stock market prediction) • How to represent more abstract data, e.g. a name? – Choose a pattern, e.g. • 0-0-1 for “Chris” • 0-1-0 for “Becky” Data Input
- 19. •Structure •Exclusive-OR •problem •Classes with •meshed regions •Most general •region shapes •Single-Layer •Two-Layer •Three-Layer •A •A•B •B •A •A•B •B •A •A•B •B •B •A •B •A •B •A Hidden Layers: Separability
- 20. •Gradient: E[w]=[E/w0,… E/wn] •(w1,w2) •(w1+w1,w2 +w2) •w=- E[w] •wi=- E/wi Learning: Gradient Descent •Chain rule:
- 21. • How can we find the right weights? By Geoffrey Hinton, Godfather of deep learning https://www.youtube.com/watch?v=qStsskXgZq8&index =3&list=PLnnr1O8OWc6arChulXyEhNvEuZcSHv2aV https://www.youtube.com/watch?v=xfPz92B0rv8&index =4&list=PLnnr1O8OWc6arChulXyEhNvEuZcSHv2aV
- 22. Recap:
- 23. Recap:
- 24. Overfitting: The downside of using powerful models
- 25. Ways to reduce overfitting
- 26. Example: Voice Recognition • Task: Learn to discriminate between two different voices saying “Hello” • Data – Sources • Steve • David – Format • Frequency distribution (60 bins) • Analogy: cochlea
- 27. • Network architecture – Feed forward network • 60 input (one for each frequency bin) • 6 hidden • 2 output (0-1 for “Steve”, 1-0 for “David”)
- 28. • Presenting the data Steve David
- 29. • Presenting the data (untrained network) Steve David 0.43 0.26 0.73 0.55
- 30. • Calculate error Steve David 0.43 – 0 = 0.43 0.26 –1 = 0.74 0.73 – 1 = 0.27 0.55 – 0 = 0.55
- 31. • Repeat process (sweep) for all training pairs – Present data – Calculate error – Backpropagate error – Adjust weights • Repeat process multiple times
- 32. • Presenting the data (trained network) Steve David 0.01 0.99 0.99 0.01
- 33. Conclusion • Learn how artificial neural network is related to biological neural network • Feedforward neural network • How weights are learned through gradient descent • Applications
- 34. From neural network to deep learning • Gradient descent will not work for deep neural network (vanishing gradients). • More effective ways to train deep networks: • Breakthrough: Unsupervised pre-training by Geoffrey Hinton, 2006. • Deep learning via Hessian-free optimization by Martens, 2010. • On the importance of initialization and momentum in deep learning, 2013. • Massive increase in computing power and massive increase in the amount of available training data.
- 35. Roadmap: Method • Tour of machine learning algorithms (1 session) • Feature engineering (1 session) – Feature selection - Yan • Supervised learning (4 sessions) – Regression models -Yan – SVM and kernel SVM - Yan – Tree-based models - Dario – Bayesian method - Xiaoyang – Ensemble models - Yan • Unsupervised learning (3 sessions) – K-means clustering – DBSCAN - Cheng – Mean shift – Agglomerative clustering – Kunal – Spectral clustering – Yan – Dimension reduction for data visualization - Yan • Deep learning (4 sessions) _ Neural network - Yan – Convolutional neural network – Hengyang Lu – Recurrent neural networks – Train deep nets with open-source tools Slides at: http://www.slideshare.net/xuyangela
- 36. Machine Learning in Rock Facies Classification An Application of XGBoost Licheng Zhang Cheng Zhan
- 37. Thank you Machine learning in Oil and Gas Conference @ Houston, April 19-20: https://energyconferencenetwork.com/machine-learning-oil-gas-2017/ 20% off, PROMO code: HML Data Disruptors Conference, ddc (energy) @ Houston, June 14 PROMO code: PROMO: HEDS49 http://www.slideshare.net/xuyangela