Natural langaugea processing n gram models
- 1. 1 CS 388: Natural Language Processing: N-Gram Language Models Raymond J. Mooney University of Texas at Austin
- 2. Language Models • Formal grammars (e.g. regular, context free) give a hard “binary” model of the legal sentences in a language. • For NLP, a probabilistic model of a language that gives a probability that a string is a member of a language is more useful. • To specify a correct probability distribution, the probability of all sentences in a language must sum to 1.
- 3. Uses of Language Models • Speech recognition – “I ate a cherry” is a more likely sentence than “Eye eight uh Jerry” • OCR & Handwriting recognition – More probable sentences are more likely correct readings. • Machine translation – More likely sentences are probably better translations. • Generation – More likely sentences are probably better NL generations. • Context sensitive spelling correction – “Their are problems wit this sentence.”
- 4. Completion Prediction • A language model also supports predicting the completion of a sentence. – Please turn off your cell _____ – Your program does not ______ • Predictive text input systems can guess what you are typing and give choices on how to complete it.
- 5. N-Gram Models • Estimate probability of each word given prior context. – P(phone | Please turn off your cell) • Number of parameters required grows exponentially with the number of words of prior context. • An N-gram model uses only N1 words of prior context. – Unigram: P(phone) – Bigram: P(phone | cell) – Trigram: P(phone | your cell) • The Markov assumption is the presumption that the future behavior of a dynamical system only depends on its recent history. In particular, in a kth-order Markov model, the next state only depends on the k most recent states, therefore an N-gram model is a (N1)-order Markov model.
- 6. N-Gram Model Formulas • Word sequences • Chain rule of probability • Bigram approximation • N-gram approximation n n w w w ... 1 1 ) | ( ) | ( )... | ( ) | ( ) ( ) ( 1 1 1 1 1 2 1 3 1 2 1 1 k n k k n n n w w P w w P w w P w w P w P w P ) | ( ) ( 1 1 1 1 k N k n k k n w w P w P ) | ( ) ( 1 1 1 k n k k n w w P w P
- 7. Estimating Probabilities • N-gram conditional probabilities can be estimated from raw text based on the relative frequency of word sequences. • To have a consistent probabilistic model, append a unique start (<s>) and end (</s>) symbol to every sentence and treat these as additional words. ) ( ) ( ) | ( 1 1 1 n n n n n w C w w C w w P ) ( ) ( ) | ( 1 1 1 1 1 1 n N n n n N n n N n n w C w w C w w P Bigram: N-gram:
- 8. Generative Model & MLE • An N-gram model can be seen as a probabilistic automata for generating sentences. • Relative frequency estimates can be proven to be maximum likelihood estimates (MLE) since they maximize the probability that the model M will generate the training corpus T. Initialize sentence with N1 <s> symbols Until </s> is generated do: Stochastically pick the next word based on the conditional probability of each word given the previous N 1 words. )) ( | ( argmax ˆ M T P
- 9. Example from Textbook • P(<s> i want english food </s>) = P(i | <s>) P(want | i) P(english | want) P(food | english) P(</s> | food) = .25 x .33 x .0011 x .5 x .68 = .000031 • P(<s> i want chinese food </s>) = P(i | <s>) P(want | i) P(chinese | want) P(food | chinese) P(</s> | food) = .25 x .33 x .0065 x .52 x .68 = .00019
- 10. Train and Test Corpora • A language model must be trained on a large corpus of text to estimate good parameter values. • Model can be evaluated based on its ability to predict a high probability for a disjoint (held-out) test corpus (testing on the training corpus would give an optimistically biased estimate). • Ideally, the training (and test) corpus should be representative of the actual application data. • May need to adapt a general model to a small amount of new (in-domain) data by adding highly weighted small corpus to original training data.
- 11. A Problem for N-Grams: Long Distance Dependencies • Many times local context does not provide the most useful predictive clues, which instead are provided by long-distance dependencies. – Syntactic dependencies • “The man next to the large oak tree near the grocery store on the corner is tall.” • “The men next to the large oak tree near the grocery store on the corner are tall.” – Semantic dependencies • “The bird next to the large oak tree near the grocery store on the corner flies rapidly.” • “The man next to the large oak tree near the grocery store on the corner talks rapidly.” • More complex models of language are needed to handle such dependencies.
- 12. Summary • Language models assign a probability that a sentence is a legal string in a language. • They are useful as a component of many NLP systems, such as ASR, OCR, and MT. • Simple N-gram models are easy to train on unsupervised corpora and can provide useful estimates of sentence likelihood. • MLE gives inaccurate parameters for models trained on sparse data. • Smoothing techniques adjust parameter estimates to account for unseen (but not impossible) events.