The Transformer - Xavier Giró - UPC Barcelona 2021
0 likes•339 views
Xavier Giro-I-Nieto's lecture on transformers outlines key mechanisms such as self-attention and multi-head self-attention, explaining their roles in natural language processing and image generation. The discussion also covers positional encodings and the removal of recurrent layers in the transformer architecture. Various references and studies are provided to support these concepts, demonstrating the broad applicability of transformers beyond language.
![6
Reminder
Attention is a mechanism to compute a context vector (c) for a query (Q) as a
weighted sum of values (V).
Figure: Nikhil Shah, “Attention? An Other Perspective! [Part 1]” (2020)](https://image.slidesharecdn.com/20transformerdlai2021-220126134021/85/The-Transformer-Xavier-Giro-UPC-Barcelona-2021-6-320.jpg)
![25
Study case: Self-Attention in for image generation
#SAGAN Zhang, Han, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. "Self-attention generative adversarial
networks." ICML 2019. [video]
Figure:
Frank Xu
Generator (G): Details can be generated using cues from all feature locations.
Discriminator: Can check consistency betweenn features in distant portions of the image.](https://image.slidesharecdn.com/20transformerdlai2021-220126134021/85/The-Transformer-Xavier-Giro-UPC-Barcelona-2021-25-320.jpg)
![26
Study case: Self-Attention in for image generation
#SAGAN Zhang, Han, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. "Self-attention generative adversarial
networks." ICML 2019. [video]
Query locations Attention maps for differet query locations](https://image.slidesharecdn.com/20transformerdlai2021-220126134021/85/The-Transformer-Xavier-Giro-UPC-Barcelona-2021-26-320.jpg)
![Multi-head Self-Attention: Visualization
32
#BertViz Vig, Jesse. "A multiscale visualization of attention in the transformer model." ACL 2019. [code] [tweet]
Each colour
corresponds
to a head.
Blue: First
head only.
Multi-color:
Multiple
heads.](https://image.slidesharecdn.com/20transformerdlai2021-220126134021/85/The-Transformer-Xavier-Giro-UPC-Barcelona-2021-32-320.jpg)
![33
Self-Attention and Convolutional Layers
Cordonnier, J. B., Loukas, A., & Jaggi, M. On the relationship between self-attention and convolutional layers. ICLR 2020.
[tweet] [code]](https://image.slidesharecdn.com/20transformerdlai2021-220126134021/85/The-Transformer-Xavier-Giro-UPC-Barcelona-2021-33-320.jpg)
![The Transformer: Layers
45
#BertViz Vig, Jesse. "A multiscale visualization of attention in the transformer model." ACL 2019. [code] [tweet]
A birds-eye view of attention across all of the model’s layers and heads](https://image.slidesharecdn.com/20transformerdlai2021-220126134021/85/The-Transformer-Xavier-Giro-UPC-Barcelona-2021-45-320.jpg)
![47
Are Transformers for Language only ? NO !!
#ViT Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa
Dehghani et al. "An image is worth 16x16 words: Transformers for image recognition at scale." ICLR 2021. [blog] [code]](https://image.slidesharecdn.com/20transformerdlai2021-220126134021/85/The-Transformer-Xavier-Giro-UPC-Barcelona-2021-47-320.jpg)
![52
Learn more
● Tutorials
○ Sebastian Ruder, Deep Learning for NLP Best Practices # Attention (2017).
○ Chris Olah, Shan Carter, “Attention and Augmented Recurrent Neural Networks”. distill.pub 2016.
○ Lilian Weg, The Transformer Family. Lil’Log 2020
● Twitter threads
○ Christian Wolf (INSA Lyon)
● Scientific publications
○ #Perceiver Jaegle, Andrew, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira.
"Perceiver: General perception with iterative attention." arXiv preprint arXiv:2103.03206 (2021).
○ #Longformer Beltagy, Iz, Matthew E. Peters, and Arman Cohan. "Longformer: The long-document transformer." arXiv
preprint arXiv:2004.05150 (2020).
○ Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F. Transformers are RNS: Fast autoregressive transformers with linear
attention. ICML 2020.
○ Siddhant M. Jayakumar, Wojciech M. Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye
Teh, Tim Harley, Razvan Pascanu, “Multiplicative Interactions and Where to Find Them”. ICLR 2020. [tweet]
○ Self-attention in language
■ Cheng, J., Dong, L., & Lapata, M. (2016). Long short-term memory-networks for machine reading. arXiv preprint
arXiv:1601.06733.
○ Self-attention in images
■ Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., Ku, A., & Tran, D. (2018). Image transformer. ICML
2018.
■ Wang, Xiaolong, Ross Girshick, Abhinav Gupta, and Kaiming He. "Non-local neural networks." In CVPR 2018.](https://image.slidesharecdn.com/20transformerdlai2021-220126134021/85/The-Transformer-Xavier-Giro-UPC-Barcelona-2021-52-320.jpg)
The Transformer - Xavier Giró - UPC Barcelona 2021
- 1. The Transformer Lecture 20 Xavier Giro-i-Nieto Associate Professor Universitat Politecnica de Catalunya @DocXavi [email protected]
- 3. 3 Acknowledgments Marta R. Costa-jussà Associate Professor Universitat Politècnica de Catalunya Carlos Escolano PhD Candidate Universitat Politècnica de Catalunya Gerard I. Gállego PhD Student Universitat Politècnica de Catalunya [email protected] @geiongallego
- 5. 5 Reminder Nikhil Sha, “Attention ? An other Perspective!”. 2020.
- 6. 6 Reminder Attention is a mechanism to compute a context vector (c) for a query (Q) as a weighted sum of values (V). Figure: Nikhil Shah, “Attention? An Other Perspective! [Part 1]” (2020)
- 7. 7 Reminder Nikhil Sha, “Attention ? An other Perspective!”. 2020.
- 8. 8 Reminder: Seq2Seq with Cross-Attention Slide concept: Abigail See, Matthew Lamm (Stanford CS224N), 2020 In this case, cross-attention refers to the attention between the encoder and decoder states.
- 9. 9 Nikhil Sha, “Attention ? An other Perspective!”. 2020. What may the term “self” refer to, as a contrast of “cross”-attention ?
- 11. 11 Self-Attention (or intra-Attention) Lin, Z., Feng, M., Santos, C. N. D., Yu, M., Xiang, B., Zhou, B., & Bengio, Y. A structured self-attentive sentence embedding. ICLR 2017. Figure: Jay Alammar, “The Illustrated Transformer” Self-attention refers to attending to other elements from the SAME sequence.
- 12. 12 Self-Attention (or intra-Attention) Nikhil Sha, “Attention ? An other Perspective!”. 2020. Query (Q) g(x) = WQ x Key (K) f(x) = WK x Value (V) h(x) = WV x WQ , WK and WV are projection layers shared across all words.
- 13. 13 Self-Attention (or intra-Attention) Which steps are necessary to compute the contextual representation of a word embedding e2 in a sequences of four words embeddings (e1 , e2 , e3 , e4 ) ? A (scaled) dot-product is computed between each pair of word embeddings (eg. e1 and e2 )... #Transformer Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I.. Attention is all you need. NeurIPS 2017.
- 14. 14 Self-Attention (or intra-Attention) Which steps are necessary to compute the contextual representation of a word embedding e2 in a sequences of four words embeddings (e1 , e2 , e3 , e4 ) ? … a softmax layer normalizes the attention scores to obtain the attention distribution...
- 15. 15 Self-Attention (or intra-Attention) Which steps are necessary to compute the contextual representation of a word embedding e2 in a sequences of four words embeddings (e1 , e2 , e3 , e4 ) ? ...the same word embeddings are combined to obtain the contextual representation e2 ’.
- 16. 16 Self-Attention (or intra-Attention) Figure: Jay Alammar, “The illustrated Transformer” (2018)
- 17. 17 Self-Attention (or intra-Attention) Figure: Jay Alammar, “The illustrated Transformer” (2018)
- 18. 18 Self-Attention (or intra-Attention) Figure: Jay Alammar, “The illustrated Transformer” (2018)
- 19. 19 Self-Attention (or intra-Attention) Figure: Jay Alammar, “The illustrated Transformer” (2018)
- 20. 20 Self-Attention (or intra-Attention) Figure: Jay Alammar, “The illustrated Transformer” (2018)
- 21. 21 Self-Attention (or intra-Attention) Figure: Jay Alammar, “The illustrated Transformer” (2018)
- 22. 22 Self-Attention (or intra-Attention) Figure: Jay Alammar, “The illustrated Transformer” (2018)
- 23. 23 Self-Attention (or intra-Attention) Figure: Jay Alammar, “The illustrated Transformer” (2018)
- 24. 24 Self-Attention (or intra-Attention) Scaled dot-product attention Figure: Jay Alammar, “The illustrated Transformer” (2018)
- 25. 25 Study case: Self-Attention in for image generation #SAGAN Zhang, Han, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. "Self-attention generative adversarial networks." ICML 2019. [video] Figure: Frank Xu Generator (G): Details can be generated using cues from all feature locations. Discriminator: Can check consistency betweenn features in distant portions of the image.
- 26. 26 Study case: Self-Attention in for image generation #SAGAN Zhang, Han, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. "Self-attention generative adversarial networks." ICML 2019. [video] Query locations Attention maps for differet query locations
- 27. Outline 1. Motivation 2. Self-attention 3. Multi-head Self-Attention (MHSA) 27
- 28. 28 Multi-Head Self-Attention (MHSA) Nikhil Sha, “Attention ? An other Perspective!”. 2020. In vanilla self-attention, a single set of projection matrices WQ , WK , WV is used.
- 29. 29 Nikhil Sha, “Attention ? An other Perspective!”. 2020. In multi-head self-attention, multiple sets of projection matrices are used, and can provide different contextual representations for the same input token. Multi-Head Self-Attention (MHSA)
- 30. 30 The multi-head self-attended E’i matrixes are concatenated: Figure: Jay Alammar, “The illustrated Transformer” (2018) Multi-Head Self-Attention (MHSA)
- 31. 31 A fully connected layer on top combines everything in a new E’. Figure: Jay Alammar, “The illustrated Transformer” (2018) Multi-Head Self-Attention (MHSA)
- 32. Multi-head Self-Attention: Visualization 32 #BertViz Vig, Jesse. "A multiscale visualization of attention in the transformer model." ACL 2019. [code] [tweet] Each colour corresponds to a head. Blue: First head only. Multi-color: Multiple heads.
- 33. 33 Self-Attention and Convolutional Layers Cordonnier, J. B., Loukas, A., & Jaggi, M. On the relationship between self-attention and convolutional layers. ICLR 2020. [tweet] [code]
- 34. Outline 1. Motivation 2. Self-attention 3. Multi-head Attention 4. Positional Encoding 34
- 35. Positional Encoding 35 Given that the attention mechanism allows accessing all input (and output) tokens, we no longer need a memory through recurrent layers.
- 36. Positional Encoding 36 Figure: Jay Alammar, “The illustrated Transformer” (2018) Where is the relative relation in the sequence encoded ?
- 37. Positional Encoding 37 Figure: Jay Alammar, “The illustrated Transformer” (2018) Where is the relative relation in the sequence encoded ?
- 38. Positional Encoding 38 Maria Ribalta, Pere-Pau Vàzquez, “Visualization is all you need”. UPC GCED 2020. Sinusoidal functions are typically used to provide positional encodings.
- 39. Positional Encoding 39 Figure: Jay Alammar, “The illustrated Transformer” (2018)
- 40. Outline 1. Motivation 2. Self-attention 3. Multi-head Attention 4. Positional Encoding 5. The Transformer 40
- 41. The Transformer 41 #Transformer Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I.. Attention is all you need. NeurIPS 2017. The Transformer removed the recurrency mechanism thanks to self-attention.
- 42. The Transformer 42 #Transformer Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I.. Attention is all you need. NeurIPS 2017. Positional Encoding over the output sequence. Positional Encoding over the input sequence. Auto-regressive (at test).
- 43. The Transformer 43 #Transformer Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I.. Attention is all you need. NeurIPS 2017. Cross-Attention (or inter-attention) between input and output tokens Self-attention for the input tokens. Self-attention for the output tokens.
- 44. The Transformer: Layers 44 #Transformer Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I.. Attention is all you need. NeurIPS 2017. N decoder layers N encoder layers
- 45. The Transformer: Layers 45 #BertViz Vig, Jesse. "A multiscale visualization of attention in the transformer model." ACL 2019. [code] [tweet] A birds-eye view of attention across all of the model’s layers and heads
- 46. The Transformer: Visualization 46 Maria Ribalta, Pere-Pau Vàzquez, “Visualization is all you need”. UPC GCED 2020.
- 47. 47 Are Transformers for Language only ? NO !! #ViT Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani et al. "An image is worth 16x16 words: Transformers for image recognition at scale." ICLR 2021. [blog] [code]
- 48. Outline 1. Motivation 2. Self-attention 3. Multi-head Attention 4. Positional Encoding 5. The Transformer 48
- 49. 49 (extra) PyTorch Lab on Google Colab DL resources from UPC Telecos: ● Lectures (with Slides & Videos) ● Labs Gerard Gallego [email protected] Student PhD Universitat Politecnica de Catalunya Technical University of Catalonia
- 50. 50 Software ● Transformers in HuggingFace. ● GPT-Neo by EleutherAI ○ Similar results to GPT-3, but smaller and open source. ● Andrej Karpathy, minGPT (2020).
- 51. 51 Learn more Ashish Vaswani, Stanford CS224N 2019.
- 52. 52 Learn more ● Tutorials ○ Sebastian Ruder, Deep Learning for NLP Best Practices # Attention (2017). ○ Chris Olah, Shan Carter, “Attention and Augmented Recurrent Neural Networks”. distill.pub 2016. ○ Lilian Weg, The Transformer Family. Lil’Log 2020 ● Twitter threads ○ Christian Wolf (INSA Lyon) ● Scientific publications ○ #Perceiver Jaegle, Andrew, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. "Perceiver: General perception with iterative attention." arXiv preprint arXiv:2103.03206 (2021). ○ #Longformer Beltagy, Iz, Matthew E. Peters, and Arman Cohan. "Longformer: The long-document transformer." arXiv preprint arXiv:2004.05150 (2020). ○ Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F. Transformers are RNS: Fast autoregressive transformers with linear attention. ICML 2020. ○ Siddhant M. Jayakumar, Wojciech M. Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, Razvan Pascanu, “Multiplicative Interactions and Where to Find Them”. ICLR 2020. [tweet] ○ Self-attention in language ■ Cheng, J., Dong, L., & Lapata, M. (2016). Long short-term memory-networks for machine reading. arXiv preprint arXiv:1601.06733. ○ Self-attention in images ■ Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., Ku, A., & Tran, D. (2018). Image transformer. ICML 2018. ■ Wang, Xiaolong, Ross Girshick, Abhinav Gupta, and Kaiming He. "Non-local neural networks." In CVPR 2018.
- 53. 53 Questions ?