Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems
- 1. 2022. 06. 10 Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems Andrea Madotto, Chien-Sheng Wu, Pascale Fung ACL 2018 Hongkyu Lim
- 2. Contents • Overview • Introduction • Model Description • Memory Encoder • Memory Decoder • Sentinel • Memory Content • Experimental Results • Analysis and Discussion • Conclusion
- 3. 3 Overview • In Task Oriented Dialog system, it is hard to combine Knowledge base(KB). • Struggling to combine KB to RNN hidden states • Time consumption : using attention mechanism • Mem2Seq is a solution to solve the issues. • Mem2Seq is a model that combines pointer network and attention.
- 4. 4 Introduction • Task oriented dialog system is used to conduct particular objectives. • It is essential to generate query with KB. • Currently(2018), RNN based on hidden states has yielded good performances. • But, there are still problems • It is hard to comprehend KB and RNN hidden states • Takes too long to process long sequences with attention
- 5. 5 Introduction • MemNN • A Recurrent attention model to utilize large external memory • Reports embedding to the external memory • Reads the memory repeatedly with query vectors • This approach enables… • Remembers KB for longer than before • Encodes long sequential dialog fast • However… • MemNN only chooses from the pool. • It does not generate answers.
- 6. 6 Model Description • Mem2Seq • Solves the limitations of MemNN • Mem2Seeq relates concepts of pointer network to multi-hop attention mechanism. • Mem2Seq copies words directly from KB • Mem2Seq learns generating dynamic query to access to memory.
- 7. 7 Model Description • Mem2Seq(architecture) • Composed of MemNN Encoder and memory decoder • MemNN Encoder makes vectors for dialog reports • Memory Decoder generates responses by reading and copying memory
- 8. 8 Model Description • Terms & Equations • Sequence Tokens for dialog records • $ is a special sign of token to generate words from memory content • Tuple for Knowledge Base • Concat of X and B
- 9. 9 Model Description • Memory Encoder • 𝑈 is a word wise concatenation of dialog and sentinel token. • The memory of MemNN is represented as • 𝐶 is a vector mapped with token used in reading query vectors. • Repeated for K hops. • For each memory sequence, the model calculates attention weights at hop k.
- 10. 10 Model Description • Memory Encoder • pk is responsible for memory selector to assign relations between memory queries. • The model reads memory ok through the sum of weights • The result of the encoder is ok and it is the input of the decoder of Mem2Seq.
- 11. 11 Model Description • Memory Decoder • Uses both dialog records and KB • GRU modules receives previously generated words and query to generate new queries every time step t. • Query h0 is the result of the Encoder • In every step, the decoder computes vocabulary distributions and memory contents distributions • The decoder generates tokens at the memory by pointing the input words.
- 12. 12 Model Description • Sentinel • If memory has no required words, memory content distribution yields sentinel words. • Memory Content • Dialog record is saved in the memory with respect to words. • Speakers and time are added to each token. • When saving KB, the token is based on subject, relations, and objects. • KB is only used to consult on particular conversations.
- 14. 14 Analysis and Discussion • Memory Attention • As shown in the picture, the distribution of weights is very clear.
- 15. 15 Conclusion • Mem2Seq is a memory to sequence model for task—oriented dialog system in end-to-end framework. • Mem2Seq is combining multi-hop attention mechanism of end-to-end memory network with pointer network. • They validated the performance of Mem2Seq with experiments.
- 16. Thank you