Textrank algorithm
15 likes13,546 views
Textrank is a graph-based method for summarization that works by (1) separating text into sentences, (2) building a sparse matrix of word counts in each sentence, (3) normalizing words with tf-idf scores, (4) constructing a similarity matrix between sentences based on shared words, and (5) using PageRank to score sentences based on their similarity to other sentences.
![1. Separate the Text into Sentences
• Apply PunktSentenceTokenizer from the Python
NLTK Library
“Hi world! Hello
world! This is
Andrew.”
[“Hi world!”, “Hello
world!”, “This is
Andrew.”]](https://image.slidesharecdn.com/textrankalgorithm-141001134151-phpapp02/85/Textrank-algorithm-3-320.jpg)
![2. Build a sparse matrix of words and
the count it appears in each sentence
[“Hi world!”, “Hello
world!”, “This is
Andrew.”]
(Sen , word) Count
(0 , 2)
(0 , 5)
(1 , 5)
(1 , 1)
(2 , 4)
(2 , 3)
(2 , 0)
1
1
1
1
1
1
1](https://image.slidesharecdn.com/textrankalgorithm-141001134151-phpapp02/85/Textrank-algorithm-4-320.jpg)
Textrank algorithm
- 1. How Does Textrank Work? Andrew Koo - Insight Data Science
- 2. Textrank • Separate the text into sentences based on a trained model • Build a sparse matrix of words and the count it appears in each sentence • Normalize each word with tf-idf • Construct the similarity matrix between sentences • Use Pagerank to score the sentences in graph
- 3. 1. Separate the Text into Sentences • Apply PunktSentenceTokenizer from the Python NLTK Library “Hi world! Hello world! This is Andrew.” [“Hi world!”, “Hello world!”, “This is Andrew.”]
- 4. 2. Build a sparse matrix of words and the count it appears in each sentence [“Hi world!”, “Hello world!”, “This is Andrew.”] (Sen , word) Count (0 , 2) (0 , 5) (1 , 5) (1 , 1) (2 , 4) (2 , 3) (2 , 0) 1 1 1 1 1 1 1
- 5. 3. Normalize each word with tf-idf • tf: term frequency - how frequent a term occurs in a document • idf: inverse doc frequency - how important a word is (weigh down the frequent terms, ex: is, does, how) (Sen , word) Count (0 , 2) (0 , 5) (1 , 5) (1 , 1) (2 , 4) (2 , 3) (2 , 0) 1 1 1 1 1 1 1 (Sen , word) Count (0 , 2) (0 , 5) (1 , 5) (1 , 1) (2 , 4) (2 , 3) (2 , 0) 0.796 0.605 0.605 0.796 0.577 0.577 0.577
- 6. 4. Construct the similarity matrix between sentences (Sen , word) Count (0 , 2) (0 , 5) (1 , 5) (1 , 1) (2 , 4) (2 , 3) (2 , 0) 0.796 0.605 0.605 0.796 0.577 0.577 0.577 1 0.366 0 0.366 1 0 0 0 1 matrix * matrix.T similarity matrix
- 7. 5. Use Pagerank to score the sentences in graph • Rank the sentences with underlying assumption that “summary sentences” are similar to most other sentences