Study of End to End memory networks
Download as PPTX, PDF4 likes1,617 views
The document is a presentation on end-to-end memory networks (MemN2N) given by Ashish Menkudale at the University of Illinois at Chicago. It discusses the architecture, advantages, and applications of MemN2N in natural language processing, highlighting its capability for reasoning and memory access through multiple lookups. Additionally, it addresses challenges regarding supervision in training and explores comparisons with traditional recurrent neural networks (RNNs).
Study of End to End memory networks
- 1. A presentation on End to End Memory Networks (MemN2N) Slides: 26 Time: 15 minutes IE 594 Data Science 2 University of Illinois at Chicago, February 2017 Under the guidance of, Prof. Dr. Ashkan Sharabiani By, Ashish Menkudale
- 2. The kitchen is north of the hallway. The bathroom is west of the bedroom. The den is east of the hallway. The office is south of the bedroom. How do you go from den to kitchen? 2
- 3. The kitchen is north of the hallway. The bathroom is west of the bedroom. The den is east of the hallway. The office is south of the bedroom. How do you go from den to kitchen? Kitchen Hallway Bathroom Bedroom Den Office West, North. West North 3
- 4. Brian is frog. Lily is grey. Brian is yellow. Julius is green. What color is Greg? Greg is frog. 4
- 5. Brian is frog. Lily is grey. Brian is yellow. Julius is green. What color is Greg? Greg is frog. Yellow. 5
- 6. External Global Memory Memory Module Controller Module Output Read Write Input Dedicated separate memory module. Memory can be stack or list/set of vectors. Control module accesses memory (read, write). Advantage: stable, scalable. Charles Babbage Invented analytical engine Concept. 1791-1871 Konrad Zuse Invented stored-program concept. 1910-1995 6
- 7. Warren Sturgis McCulloch. Computational model for neural networks 1898-1969 Memory Networks • Memory network with large external memory. required for low level tasks like object recognition. • Writes everything to the memory, but reads only relative information. • Attempts to add long term memory component to make it more like artificial intelligence. • Two types: • Strongly supervised memory network: Hard addressing. • Weekly supervised memory network: Soft addressing. • Hard addressing: max of the inner product between the internal state and memory contents. Mary is in garden. John is in office. Q: Where is John? Bob is in kitchen. Walter Pitts. Computational model for neural networks 1923-1969 7
- 8. Memory Vectors Example: Constructing memory vectors with bag of words (BoW) Embed each word Sum embedding vectors “Sam drops apple” 𝑉 𝑆𝑎𝑚 + 𝑉 𝑑𝑟𝑜𝑝𝑠 + 𝑉apple = 𝑚 Embedding vectors memory vector Example: Temporal structure – special words for time and include them in bag of words. 1. Sam moved to garden 2. Sam moved to kitchen. 3. Sam drops apple. 𝑉 𝑆𝑎𝑚 + 𝑉 𝑑𝑟𝑜𝑝𝑠 + 𝑉apple + 𝑉time = 𝑚 Time EmbeddingTime Stamp 8
- 9. Bob is in kitchen. Mary is in garden. John is in office. Where is John? Embed Embed Embed Embed X X X Max Internal State vector John is in office Embed + Decoder Office Output Memory Controller Memory Networks Input 9
- 10. Issues with Memory Network • Requires explicit supervision of attention during training. Need to say which memory the model should use. • Need a model that just requires supervision at output. No supervision of attention required. • Only feasible for simple tasks. Severely limits application of model. 10
- 11. End-To-End Memory Networks • Soft attention version of MemN2N. • Flexible read-only memory. • Multiple memory lookups (hops). • Can consider multiple memory before deciding output. • More reasoning power. • End-to-end training. • Only needs final output for training. • Simple back-propagation. 11 Sainbayar Sukhbaatar Arthur Szlam Jason Weston Rob Fergus
- 12. Tanh / ReLU Dot product Softmax Weighted Sum Memory content Sum Linear State State Memory module Output Target Loss Function Input Controller module E.g. RNN MemN2N architecture 12
- 13. MemN2N in action : Single memory lookup Sentences {Xi} Softmax Question q Embedding BInner product Embedding A Embedding C Probability Weighted Sum ∑ InputsWeightsOutput O u Weight Softmax Predicted Answer Mary is in garden. John is in office. Bob is in kitchen. Where is John? Office Training: estimate embedding matrices A, B & C and output matrix W. 13
- 14. A3 C3 A2 C2 C1 Multiple Memory Lookups: Multiple Hops Sentences {Xi} Input 1 Output 1 ∑ Question q Input 2 Output 2 ∑ Input 3 Output 3 ∑ A1 O1 O1 O3 W 𝐴 Predicted Answer u1 u2 u3 14
- 15. Components 15 I (Input): No conversion keep original text X. G (Generalization): Stores I (X) in next available memory slot. O (Output): Loops over all memories. Find best match of 𝑚i with X. Find best match of 𝑚j with (𝑚i , X) Can be extended to multiple number of hops. R (Response) : Ranks all words in dictionary given o and returns best single word. Infact, RNN can be used here for better sentence correction.
- 16. Weight Tying 16 Weight tying : Indicates how weight vectors are multiplied to input and output component. Two Methods: Adjacent: Similar to stack layers Output embedding of one layer are input embedding of the next layer. Layer wise: Input embedding remains the same for every layer in architecture.
- 17. Scoring function 17 Question : Answers are mapped to story using word embedding. Word Embedding : Maps different words in low dimensional vector space with advantage to calculate distance between word vectors. Allow us to find similarity score between different sentence to understand maximum correlation between them. Match (‘Where is football?’, ‘John picked up the football’). qTUTUd : This model is default word embedding used in memory networks. q – Question. U – matrix by which word embedding are obtained. d – Answer.
- 18. Model Selection 18 Model Selection: Determines how to model story, questions and answer vectors for word embedding. Two possible approach: Bag of words model: Considers each word in a sentence. Embeds each word and sums resulting vector. Does not take into account context for each word. Position Encoding: Considers position/context of sentence/words. Takes care of preceding and forwarding words. Maps it to low dimensional vector space. Model Refining Addition of noise. increasing training dataset.
- 19. Decisions for Configuration 19 • Number of hops • Number of epochs • Embedding size • Training dataset • Validation dataset • Model selection • Weight tying
- 20. RNN viewpoint of MemN2N Plain RNN Memory Network RNN Input Sequence Memory RNN All Input Selected input Addressing signal Inputs are fed to RNN one-by-one in order. RNN has only one chance to look at a certain input symbol. Place all inputs in the memory. Let the model decide which part it reads next. 20
- 21. • More generic input format • Any set of vectors can be input • Each vector can be o BOW of symbols (including location) o Image feature + feature position • Location can be 1D, 2D, … • Variable size Advantages of MemN2N over RNN • Out-of-order access to input data • Less distracted by unimportant inputs • Longer term memorization • No vanishing or exploding gradient problems 21
- 22. bAbi Project: Task CAtegories Training dataset: 1000 questions for each tasks. Testing dataset: 1000 questions for each tasks.
- 23. 23 Demo for bAbi tasks
- 24. 24 bAbi Project: Benchmark results
- 25. 1. GitHub project archives: https://github.com/vinhkhuc/MemN2N-babi-python 2. https://www.msri.org/workshops/796/schedules/20462/documents/2704/assets/24734 3. Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes: https://arxiv.org/pdf/1607.00036.pdf 4. bAbi answers: https://arxiv.org/pdf/1502.05698.pdf 5. Memory Networks by Microsoft research: https://www.youtube.com/watch?v=ZwvWY9Yy76Q&t=1s 6. Memory Networks (Jenil Shah): https://www.youtube.com/watch?v=BN7Kp0JD04o 7. N gram – SVM – generative models difference. http://stackoverflow.com/questions/20315897/n-grams-vs-other- classifiers-in-text-categorization 8. Paper on results for bAbi tasks by Facebook AI team: https://papers.nips.cc/paper/5846-end-to-end-memory- networks.pdf 9. Towards AI-complete question answering : a set of prerequisite toy tasks https://arxiv.org/pdf/1502.05698.pdf 25 References
- 26. 26 Questions