![[Kim+ ICML2012] Dirichlet Process
with Mixed Random Measures : A
Nonparametric Topic Model for
Labeled Data
2012/07/28
Nakatani Shuyo @ Cybozu Labs, Inc
twitter : @shuyo](https://image.slidesharecdn.com/dp-mrmkimicml2012-120727233419-phpapp01/85/Kim-ICML2012-Dirichlet-Process-with-Mixed-Random-Measures-A-Nonparametric-Topic-Model-for-Labeled-Data-1-320.jpg)
![LDA(Latent Dirichlet Allocation)
[Blei+ 03]
• Unsupervised Topic Model
– Each word has an unobserved topic
• Parametric
– The topic size K is given in advance
via Wikipedia](https://image.slidesharecdn.com/dp-mrmkimicml2012-120727233419-phpapp01/85/Kim-ICML2012-Dirichlet-Process-with-Mixed-Random-Measures-A-Nonparametric-Topic-Model-for-Labeled-Data-2-320.jpg)
![Labeled LDA [Ramage+ 09]
• Supervised Topic Model
– Each document has an observed label
• Parametric
via [Ramage+ 09]](https://image.slidesharecdn.com/dp-mrmkimicml2012-120727233419-phpapp01/85/Kim-ICML2012-Dirichlet-Process-with-Mixed-Random-Measures-A-Nonparametric-Topic-Model-for-Labeled-Data-3-320.jpg)
![Generative Process for L-LDA
• 𝜷 𝑘 ~Dir 𝜼
topics corresponding to
𝑑 observed labels
• Λ 𝑘 ~Bernoulli Φ 𝑘
• 𝜽 𝑑 ~Dir 𝜶 𝑑
restricted to labeled
– where 𝜶 𝑑 = 𝛼𝑘 parameters
𝑑
𝑘 Λ 𝑘 =1
𝑑 𝑑
• 𝑧 𝑖 ~Multi 𝜽
𝑑
• 𝑤𝑖 ~Multi 𝜷 𝑧 𝑑
𝑖
via [Ramage+ 09]](https://image.slidesharecdn.com/dp-mrmkimicml2012-120727233419-phpapp01/85/Kim-ICML2012-Dirichlet-Process-with-Mixed-Random-Measures-A-Nonparametric-Topic-Model-for-Labeled-Data-4-320.jpg)
![Pros/Cons of L-LDA
• Pros
– Easy to implement
• Cons via [Ramage+ 09]
– It is necessary to specify label-topic
correspondence manually
• Its performance depends on the corresponds
※) My implementation is here : https://github.com/shuyo/iir/blob/master/lda/llda.py](https://image.slidesharecdn.com/dp-mrmkimicml2012-120727233419-phpapp01/85/Kim-ICML2012-Dirichlet-Process-with-Mixed-Random-Measures-A-Nonparametric-Topic-Model-for-Labeled-Data-5-320.jpg)
![DP-MRM [Kim+ 12]
– Dirichlet Process with Mixed Random Measures
• Supervised Topic Model
• Nonparametric
– K is not the topic size, but the label size
𝛼
𝑁𝑗
𝐻 𝐺0𝑘 𝐺𝑗 𝜃 𝑗𝑖 𝑥 𝑗𝑖
𝜆j 𝑟𝑗 𝐷
𝛽 𝛾𝑘 𝜂
𝐾](https://image.slidesharecdn.com/dp-mrmkimicml2012-120727233419-phpapp01/85/Kim-ICML2012-Dirichlet-Process-with-Mixed-Random-Measures-A-Nonparametric-Topic-Model-for-Labeled-Data-6-320.jpg)
![Experiments
• DP-MRM gives label-topic probabilistic
corresponding automatically.
via [Kim+ 12]](https://image.slidesharecdn.com/dp-mrmkimicml2012-120727233419-phpapp01/85/Kim-ICML2012-Dirichlet-Process-with-Mixed-Random-Measures-A-Nonparametric-Topic-Model-for-Labeled-Data-12-320.jpg)
![via [Kim+ 12]
• L-LDA can also predict single labeled document to
assign a common second label to any documents.](https://image.slidesharecdn.com/dp-mrmkimicml2012-120727233419-phpapp01/85/Kim-ICML2012-Dirichlet-Process-with-Mixed-Random-Measures-A-Nonparametric-Topic-Model-for-Labeled-Data-13-320.jpg)
![References
• [Kim+ ICML2012] Dirichlet Process with Mixed
Random Measures : A Nonparametric Topic
Model for Labeled Data
• [Ramage+ EMNLP2009] Labeled LDA : A
supervised topic model for credit attribution in
multi-labeled corpora
• [Blei+ 2003] Latent Dirichlet Allocation](https://image.slidesharecdn.com/dp-mrmkimicml2012-120727233419-phpapp01/85/Kim-ICML2012-Dirichlet-Process-with-Mixed-Random-Measures-A-Nonparametric-Topic-Model-for-Labeled-Data-14-320.jpg)
[Kim+ ICML2012] Dirichlet Process with Mixed Random Measures : A Nonparametric Topic Model for Labeled Data
- 1. [Kim+ ICML2012] Dirichlet Process with Mixed Random Measures : A Nonparametric Topic Model for Labeled Data 2012/07/28 Nakatani Shuyo @ Cybozu Labs, Inc twitter : @shuyo
- 2. LDA(Latent Dirichlet Allocation) [Blei+ 03] • Unsupervised Topic Model – Each word has an unobserved topic • Parametric – The topic size K is given in advance via Wikipedia
- 3. Labeled LDA [Ramage+ 09] • Supervised Topic Model – Each document has an observed label • Parametric via [Ramage+ 09]
- 4. Generative Process for L-LDA • 𝜷 𝑘 ~Dir 𝜼 topics corresponding to 𝑑 observed labels • Λ 𝑘 ~Bernoulli Φ 𝑘 • 𝜽 𝑑 ~Dir 𝜶 𝑑 restricted to labeled – where 𝜶 𝑑 = 𝛼𝑘 parameters 𝑑 𝑘 Λ 𝑘 =1 𝑑 𝑑 • 𝑧 𝑖 ~Multi 𝜽 𝑑 • 𝑤𝑖 ~Multi 𝜷 𝑧 𝑑 𝑖 via [Ramage+ 09]
- 5. Pros/Cons of L-LDA • Pros – Easy to implement • Cons via [Ramage+ 09] – It is necessary to specify label-topic correspondence manually • Its performance depends on the corresponds ※) My implementation is here : https://github.com/shuyo/iir/blob/master/lda/llda.py
- 6. DP-MRM [Kim+ 12] – Dirichlet Process with Mixed Random Measures • Supervised Topic Model • Nonparametric – K is not the topic size, but the label size 𝛼 𝑁𝑗 𝐻 𝐺0𝑘 𝐺𝑗 𝜃 𝑗𝑖 𝑥 𝑗𝑖 𝜆j 𝑟𝑗 𝐷 𝛽 𝛾𝑘 𝜂 𝐾
- 7. Generative Process for DP-MRM 𝛼 Each label has a random measure as topic space 𝑁𝑗 𝐻 𝐺0𝑘 𝐺𝑗 𝜃 𝑗𝑖 𝑥 𝑗𝑖 • 𝐻 = Dir 𝛽 𝜆j 𝑟𝑗 𝐷 • 𝐺0𝑘 ~DP 𝛾 𝑘 , 𝐻 𝛽 𝐾 𝛾𝑘 𝜂 • 𝜆 𝑗 ~Dir 𝒓 𝑗 𝜂 where 𝒓 𝑗 = 𝐼 𝑘∈label 𝑗 • 𝐺 𝑗 ~DP 𝛼, 𝑘∈label 𝑗 𝜆 𝑗𝑘 𝐺0𝑘 mixed random measures • 𝜃 𝑗𝑖 ~𝐺 𝑗 , 𝑥 𝑗𝑖 ~𝐹 𝜃 𝑗𝑖 = Multi 𝜃 𝑗𝑖
- 8. Stick Breaking Process • 𝑣 𝑙 𝑘 ~Beta 1, 𝛾 𝑘 , 𝜋 𝑙𝑘 = 𝑣 𝑙 𝑘 𝑙−1 𝑑=0 1 − 𝑣 𝑑𝑘 • 𝜙 𝑙𝑘 ~𝐻, 𝐺0𝑘 = ∞ 𝑙=0 𝜋 𝑙𝑘 𝛿 𝜙 𝑘 𝑙 𝑡−1 • 𝜆 𝑗 ~Dir 𝒓 𝑗 𝜂 , 𝑤 𝑗𝑡 ~Beta 1, 𝛼 , 𝜋 𝑗𝑡 = 𝑤 𝑗𝑡 𝑑=0 1 − 𝑤 𝑗𝑑 𝑘 𝑗𝑡 ∞ • 𝑘 𝑗𝑡 ~Multi 𝜆 𝑗 , 𝜓 𝑗𝑡 ~𝐺0 , 𝐺𝑗 = 𝑡=0 𝜋 𝑗𝑡 𝛿 𝜓 𝑗𝑡
- 9. Chinese Restaurant Franchise • 𝑡 𝑗𝑖 : table index of 𝑖-th term in 𝑗-th document • 𝑘 𝑗𝑡 , 𝑙 𝑗𝑡 : dish indexes on 𝑡-th table of 𝑗-th document This layer consists on only a single DP G0 on normal HDP
- 10. Inference (1) • Sampling 𝑡
- 11. Inference (2) • Sampling 𝑘 and 𝑙
- 12. Experiments • DP-MRM gives label-topic probabilistic corresponding automatically. via [Kim+ 12]
- 13. via [Kim+ 12] • L-LDA can also predict single labeled document to assign a common second label to any documents.
- 14. References • [Kim+ ICML2012] Dirichlet Process with Mixed Random Measures : A Nonparametric Topic Model for Labeled Data • [Ramage+ EMNLP2009] Labeled LDA : A supervised topic model for credit attribution in multi-labeled corpora • [Blei+ 2003] Latent Dirichlet Allocation