Neural Networks with Focus on Language Modeling
Download as PPTX, PDF3 likes440 views
Neural networks can be used for language modeling by assigning probabilities to sequences of words. Language models are evaluated based on how well they can predict upcoming words. Recurrent neural networks like LSTMs are commonly used for language modeling and have achieved better performance than traditional n-gram models, with state-of-the-art LSTMs obtaining a perplexity score of 23 compared to 41 for regular RNNs. Convolutional networks can also be applied to language modeling by treating word embeddings as inputs.
![N-gram models
Data sparsity
Lack of generalization:
[ride a horse], [ride a llama]
[ride a zebra]](https://image.slidesharecdn.com/neuralnets-180302071050/85/Neural-Networks-with-Focus-on-Language-Modeling-21-320.jpg)
![One-hot encoding
V = {zebra, horse, school, summer}
V(zebra) = [1,0,0,0]
V(Horse) = [0,1,0,0]
V(school) = [0,0,1,0]
V(summer) = [0,1,0,1]
One hot encoding is simple and yet word similarity is undefined](https://image.slidesharecdn.com/neuralnets-180302071050/85/Neural-Networks-with-Focus-on-Language-Modeling-22-320.jpg)
Neural Networks with Focus on Language Modeling
- 1. Neural Network Language Modeling ADEL RAHIMI
- 2. Table of Contents •Introduction to neural networks •Neurons •Activation Functions •Types of neural networks •Learning in neural networks •Applications of neural networks •Advantages of neural networks •Disadvantages of neural networks •Introduction to Language modeling •Neural Networks Language Modeling
- 3. Neural Networks Computing based on interaction of multiple connected processing elements Powerful in doing many processes Ability to adapt and learn Ability to deal with incomplete data
- 4. Basics of neural networks Developed from biological approaches Developed in 1943 Has one or more layers Has different types of feeding: including feedforward and feedback networks
- 6. Perceptron A mathematical model of biological neuron
- 7. Neural network neurons Receives input Multiplies input by weight applies activation function to the sum of results Output results
- 8. Neural network advantages Neural networks can perform better than normal linear programs As neural networks are parallel they can continue without failing Can be implemented in variety of applications Ability to derive meaning from complicated or imprecise data
- 9. Disadvantages Neural networks need training data Requires high processing time
- 10. Activation function Control the activity of the unit Threshold function outputs 1 when it is active and 0 when it is inactive Some examples of activation functions: Sigmoid = 1 / (1 + e-x) Tanh = 2/(1+e-2x)-1 (-1, 1)
- 11. Types of neural networks Connection type: ◦ Static ◦ Dynamic Typology: ◦ Single layer ◦ Multiple layers ◦ Recurrent Learning method: ◦ Supervised ◦ Unsupervised ◦ Reinforcement
- 12. Connections Static: Like feedforward networks Dynamic: Feedback networks; which their connection changes when the network is going backwards
- 13. Topology Single layer Multilayer Recurrent
- 14. Learning methods Supervised learning ◦ Each learning pattern: input + desired output ◦ At each presentation: adapts weights ◦ After many epochs convergence to a local minimum
- 15. Unsupervised learning No help from outside Learning by doing Pick out structures in the input: ◦ Clustering ◦ Dimensionality reduction
- 16. Reinforcement learning Inspired by behaviorist psychologists Teacher: Training data The teacher scores the performance of the training examples Use performance to shuffle weights randomly Relatively slow in learning due to randomness Examples: Robotic tasks
- 17. Neural network applications Pattern recognition Investments analysis Control systems and monitoring Mobile computing Natural language processing Forecasting sales, market, meteorology
- 18. Language modeling Filtering out bad sentences Model the sentences via probability distribution over sequences of words: Pr(w1,w2,…,wn) Assign a probability to a given sentence: S1 = “The cat jumped over the dog”, Pr(S1) ~1 S2 = “The Over cat dog jumped”, Pr(S2) ~0
- 19. Language modeling applications Machine translation: ◦ P(high winds tonight) > P(large winds tonight) Spell correction: ◦ The office is about fifteen minuets from my house ◦ P(about fifteen minutes from) > P(about fifteen minuets from) Speech Recognition: ◦ P(I saw a van) > P(eyes awe of an) Summarization, question-answering, etc.
- 20. N-gram models A sentence s = (x1, x2, ... , xT) How likely is s? p(x1, x2, ... , xT)
- 21. N-gram models Data sparsity Lack of generalization: [ride a horse], [ride a llama] [ride a zebra]
- 22. One-hot encoding V = {zebra, horse, school, summer} V(zebra) = [1,0,0,0] V(Horse) = [0,1,0,0] V(school) = [0,0,1,0] V(summer) = [0,1,0,1] One hot encoding is simple and yet word similarity is undefined
- 23. Distributional Representation Stars: shining, bright, light, dark Stars: cucumber, computer, lava, Harvard
- 24. Distributional Representation It is simple It has the notion of word similarity But it can be computationally and memory-wise inefficient
- 25. Distributional Representation V is a vocabulary wi ∈ V v(wi) ∈ Rn v(wi) is a low-dimensional, learnable, dense word vector
- 27. Deep learning + language modeling Traditionally uses architecture such as Recurrent neural networks Advancements In neural networks: LSTM LSTM is a recurrent network that can be part of an eventually bigger recurrent network. LSTM contains: Cell, input gate, output gate, forget gate The Cell is the memory, the three other gates are conventional artificial neurons
- 29. Neural Network Language Model
- 30. Neural Network Language Model
- 31. Convolutional networks CNNs are breakthrough in image processing; however they can used in language modeling as well.
- 32. Convolutional networks The input of convolutional neural nets for language modelng could either be word2vec to one- hot embedding.
- 33. Evaluation Based on: Jozefowicz, Rafal, et al. "Exploring the limits of language modeling." arXiv preprint arXiv:1602.02410 (2016). Best performance of LSTM in language modeling yields 23 in perplexity and RNN resulted 41 in perplexity.