A Simple Stochastic Gradient Variational Bayes for the Correlated Topic Model
Download as PPTX, PDF•1 like•5,267 views
This document presents a new method for estimating the posterior distribution of the Correlated Topic Model (CTM) using Stochastic Gradient Variational Bayes (SGVB). The CTM is an extension of LDA that models correlations between topics. The proposed method approximates the true posterior of the CTM with a factorial variational distribution and uses SGVB to maximize the evidence lower bound. This allows incorporating randomness into posterior inference for the CTM without requiring explicit inversion of the covariance matrix. Perplexity results on several datasets were comparable to LDA. Future work could explore online learning for topic models using neural networks.
![Contribution
•We propose a new posterior estimation for
the correlated topic model (CTM) [Blei+ 07],
•an extension of LDA [Blei+ 03] for modeling
topic correlations,
•with stochastic gradient variational Bayes
(SGVB) [Kingma+ 14].
4](https://image.slidesharecdn.com/masadaapweb2016-161204134820/85/A-Simple-Stochastic-Gradient-Variational-Bayes-for-the-Correlated-Topic-Model-4-320.jpg)
![LDA [Blei+ 03]
•Clustering word tokens by assigning each word token to
one among the 𝐾 topics.
• 𝑧 𝑑𝑖: To which topic is the 𝑖-th word token in document 𝑑 is
assigned?
• 𝜃 𝑑𝑘: How often is the topic 𝑘 talked about in document 𝑑?
• Multinomial distribution for each 𝑑
• 𝜙 𝑘𝑣: How often is the word 𝑣 used to talk about the topic 𝑘?
• Multinomial distribution for each 𝑘
discrete variables
continuous variables
5](https://image.slidesharecdn.com/masadaapweb2016-161204134820/85/A-Simple-Stochastic-Gradient-Variational-Bayes-for-the-Correlated-Topic-Model-5-320.jpg)
![CTM [Blei+ 05]
•Clustering word tokens by assigning each word token to
one among the 𝐾 topics.
• 𝑧 𝑑𝑖: To which topic is the 𝑖-th word token in document 𝑑 is
assigned?
• 𝜃 𝑑𝑘: How often is the topic 𝑘 talked about in document 𝑑?
• 𝜽 𝑑 = 𝑓 𝜼 𝑑 where 𝜼 𝑑~𝑁 𝝁, 𝚺 (logistic normal distribution)
• 𝜙 𝑘𝑣: How often is the word 𝑣 used to talk about the topic 𝑘?
• Multinomial distribution for each 𝑘
discrete variables
continuous variables
6](https://image.slidesharecdn.com/masadaapweb2016-161204134820/85/A-Simple-Stochastic-Gradient-Variational-Bayes-for-the-Correlated-Topic-Model-6-320.jpg)
![SGVB
[Kingma+ 14]
•SGVB (stochastic gradient variational Bayes) is a
general framework for estimating ELBO in
VB.
•SGVB is only applicable to continuous
distributions 𝑞 𝚯 .
•Monte Carlo integration for expectation
9](https://image.slidesharecdn.com/masadaapweb2016-161204134820/85/A-Simple-Stochastic-Gradient-Variational-Bayes-for-the-Correlated-Topic-Model-9-320.jpg)
A Simple Stochastic Gradient Variational Bayes for the Correlated Topic Model
- 1. A Simple SGVB (Stochastic Gradient Variational Bayes) for the CTM (Correlated Topic Model) Tomonari MASADA (正田备也) Nagasaki University (长崎大学) [email protected] APWeb 2016 @ Suzhou
- 2. Aim •Make an informative summary of large document sets by •extracting word lists, each relating to a different and particular topic. Topic modeling 2
- 4. Contribution •We propose a new posterior estimation for the correlated topic model (CTM) [Blei+ 07], •an extension of LDA [Blei+ 03] for modeling topic correlations, •with stochastic gradient variational Bayes (SGVB) [Kingma+ 14]. 4
- 5. LDA [Blei+ 03] •Clustering word tokens by assigning each word token to one among the 𝐾 topics. • 𝑧 𝑑𝑖: To which topic is the 𝑖-th word token in document 𝑑 is assigned? • 𝜃 𝑑𝑘: How often is the topic 𝑘 talked about in document 𝑑? • Multinomial distribution for each 𝑑 • 𝜙 𝑘𝑣: How often is the word 𝑣 used to talk about the topic 𝑘? • Multinomial distribution for each 𝑘 discrete variables continuous variables 5
- 6. CTM [Blei+ 05] •Clustering word tokens by assigning each word token to one among the 𝐾 topics. • 𝑧 𝑑𝑖: To which topic is the 𝑖-th word token in document 𝑑 is assigned? • 𝜃 𝑑𝑘: How often is the topic 𝑘 talked about in document 𝑑? • 𝜽 𝑑 = 𝑓 𝜼 𝑑 where 𝜼 𝑑~𝑁 𝝁, 𝚺 (logistic normal distribution) • 𝜙 𝑘𝑣: How often is the word 𝑣 used to talk about the topic 𝑘? • Multinomial distribution for each 𝑘 discrete variables continuous variables 6
- 7. Variational Bayes Maximization of ELBO (evidence lower bound) •VB (variational Bayes) approximates the true posterior. •An approximate posterior is introduced when ELBO is obtained by Jensen's inequality: • 𝒛: discrete hidden variables (topic assignments) • 𝚯: continuous hidden variables (multinomial parameters) 7 log evidence approximate posterior 𝑞(𝒛, 𝚯)
- 8. Factorization assumption •We assume the approximate posterior 𝑞 𝒛, 𝚯 factorizes as 𝑞 𝒛 𝑞 𝚯 . •Then ELBO can be written as 8 ×discrete continuous
- 9. SGVB [Kingma+ 14] •SGVB (stochastic gradient variational Bayes) is a general framework for estimating ELBO in VB. •SGVB is only applicable to continuous distributions 𝑞 𝚯 . •Monte Carlo integration for expectation 9
- 10. Reparameterization •We use the diagonal logistic normal for approximating the true posterior of 𝜽 𝑑. •We can efficiently sample from the logistic normal with reparameterization. 10
- 11. Monte Carlo integration •ELBO is estimated with a sample from the approximate posterior. • The discrete part 𝑞 𝒛 is estimated as in the original VB. 11
- 12. Parameter updates No explicit inversion (only Cholesky factorization) 12
- 13. "Stochastic" gradient •The expectation integrations are estimated by Monte Carlo method. •The derivatives of ELBO depend on samples. •Randomness is incorporated into the maximization of ELBO. •Does this make it easier to avoid local minima? 13
- 14. Data sets # docs # word types NYT 149,890 46,650 MOVIE 27,859 62,408 NSF 128,818 21,471 MED 125,490 42,83014
- 19. Conclusion •We incorporate randomness into the posterior inference for the CTM by using SGVB. •The proposed method gives perplexities comparable to those achieved by LDA. 19
- 20. Pro/Con •No explicit inversion of covariance matrix is required. •Careful tuning of gradient descent seems required. •Only Adam was tested. 20
- 21. Future work •Online learning for topic models with NN •NN may achieve a better approximate posterior. •SGVB can be used to estimate ELBO in a similar manner. •Document batches can be fed to VB indefinitely. •Topic word lists are then updated indefinitely. 21