Preface

• Who this book is for
• What this book covers
• To get the most out of this book
• Get in touch

1. What are Transformers?

• The ecosystem of transformers
• Optimizing NLP models with transformers
• What resources should we use?
• Summary
• Questions
• References

2. Getting Started with the Architecture of the Transformer Model FREE CHAPTER

• The rise of the Transformer: Attention is All You Need
• Training and performance
• Tranformer models in Hugging Face
• Summary
• Questions
• References

3. Fine-Tuning BERT Models

• The architecture of BERT
• Fine-tuning BERT
• Summary
• Questions
• References

4. Pretraining a RoBERTa Model from Scratch

• Training a tokenizer and pretraining a transformer
• Building KantaiBERT from scratch
• Next steps
• Summary
• Questions
• References

5. Downstream NLP Tasks with Transformers

• Transduction and the inductive inheritance of transformers
• Transformer performances versus Human Baselines
• Running downstream tasks
• Summary
• Questions
• References

6. Machine Translation with the Transformer

• Defining machine translation
• Preprocessing a WMT dataset
• Evaluating machine translation with BLEU
• Translation with Google Translate
• Translations with Trax
• Summary
• Questions
• References

7. The Rise of Suprahuman Transformers with GPT-3 Engines

• Suprahuman NLP with GPT-3 transformer models
• The architecture of OpenAI GPT transformer models
• Generic text completion with GPT-2
• Training a custom GPT-2 language model
• Running OpenAI GPT-3 tasks
• Comparing the output of GPT-2 and GPT-3
• Fine-tuning GPT-3
• The role of an Industry 4.0 AI specialist
• Summary
• Questions
• References

8. Applying Transformers to Legal and Financial Documents for AI Text Summarization

• Designing a universal text-to-text model
• Text summarization with T5
• Summarization with GPT-3
• Summary
• Questions
• References

9. Matching Tokenizers and Datasets

• Matching datasets and tokenizers
• Standard NLP tasks with specific vocabulary
• Exploring the scope of GPT-3
• Summary
• Questions
• References

10. Semantic Role Labeling with BERT-Based Transformers

• Getting started with SRL
• SRL experiments with the BERT-based model
• Basic samples
• Difficult samples
• Questioning the scope of SRL
• Summary
• Questions
• References

11. Let Your Data Do the Talking: Story, Questions, and Answers

• Methodology
• Method 0: Trial and error
• Method 1: NER first
• Method 2: SRL first
• Next steps
• Summary
• Questions
• References

12. Detecting Customer Emotions to Make Predictions

• Getting started: Sentiment analysis transformers
• The Stanford Sentiment Treebank (SST)
• Predicting customer behavior with sentiment analysis
• Sentiment analysis with GPT-3
• Some Pragmatic I4.0 thinking before we leave
• Summary
• Questions
• References

13. Analyzing Fake News with Transformers

• Emotional reactions to fake news
• A rational approach to fake news
• Before we go
• Summary
• Questions
• References

14. Interpreting Black Box Transformer Models

• Transformer visualization with BertViz
• LIT
• Transformer visualization via dictionary learning
• Exploring models we cannot access
• Summary
• Questions
• References

15. From NLP to Task-Agnostic Transformer Models

• Choosing a model and an ecosystem
• The Reformer
• DeBERTa
• From Task-Agnostic Models to Vision Transformers
• An expanding universe of models
• Summary
• Questions
• References

16. The Emergence of Transformer-Driven Copilots

• Prompt engineering
• Copilots
• Domain-specific GPT-3 engines
• Transformer-based recommender systems
• Computer vision
• Humans and AI copilots in metaverses
• Summary
• Questions
• References

17. The Consolidation of Suprahuman Transformers with OpenAI’s ChatGPT and GPT-4

• Consolidating suprahuman NLP with ChatGPT and GPT-4 transformer models
• Jump-starting the ChatGPT API
• ChatGPT Plus writes and comments on a program
• Getting started with the GPT-4 API
• Advanced prompt engineering
• Explainable AI (XAI)
• Getting started with the DALL-E 2 API
• Putting it all together
• Summary
• Questions
• References

18. Other Books You May Enjoy

19. Index

Appendix I — Terminology of Transformer Models

• Stack
• Sublayer
• Attention heads

1. Appendix II — Hardware Constraints for Transformer Models

• The Architecture and Scale of Transformers
• Why GPUs are so special
• GPUs are designed for parallel computing
• GPUs are also designed for matrix multiplication
• Implementing GPUs in code
• Testing GPUs with Google Colab
• Google Colab Free with a CPU
• Google Colab Pro with a GPU

2. Appendix III — Generic Text Completion with GPT-2

• Step 1: Activating the GPU
• Step 2: Cloning the OpenAI GPT-2 repository
• Step 3: Installing the requirements
• Step 4: Checking the version of TensorFlow
• Step 5: Downloading the 345M-parameter GPT-2 model
• Steps 6-7: Intermediate instructions
• Steps 7b-8: Importing and defining the model
• Step 9: Interacting with GPT-2
• References

3. Appendix IV — Custom Text Completion with GPT-2

• Training a GPT-2 language model
• References

4. Appendix V — Answers to the Questions

• Chapter 2, Getting Started with the Architecture of the Transformer Model
• Chapter 3, Fine-Tuning BERT Models
• Chapter 4, Pretraining a RoBERTa Model from Scratch
• Chapter 5, Downstream NLP Tasks with Transformers
• Chapter 6, Machine Translation with the Transformer
• Chapter 7, The Rise of Suprahuman Transformers with GPT-3 Engines
• Chapter 8, Applying Transformers to Legal and Financial Documents for AI Text Summarization
• Chapter 9, Matching Tokenizers and Datasets
• Chapter 10, Semantic Role Labeling with BERT-Based Transformers
• Chapter 11, Let Your Data Do the Talking: Story, Questions, and Answers
• Chapter 12, Detecting Customer Emotions to Make Predictions
• Chapter 13, Analyzing Fake News with Transformers
• Chapter 14, Interpreting Black Box Transformer Models
• Chapter 15, From NLP to Task-Agnostic Transformer Models
• Chapter 16, The Emergence of Transformer-Driven Copilots
• Chapter 17, The Consolidation of Suprahuman Transformers with OpenAI’s ChatGPT and GPT-4