Preface

• Why I wrote this book
• The approach
• Who this book is for
• What this book covers
• To get the most out of this book
• Get in touch

1. What Is Reinforcement Learning?

• Supervised learning
• Unsupervised learning
• Reinforcement learning
• RL's complications
• RL formalisms
• The theoretical foundations of RL
• Summary

2. OpenAI Gym FREE CHAPTER

• The anatomy of the agent
• Hardware and software requirements
• The OpenAI Gym API
• The random CartPole agent
• Extra Gym functionality – wrappers and monitors
• Summary

3. Deep Learning with PyTorch

• Tensors
• Gradients
• NN building blocks
• Custom layers
• The final glue – loss functions and optimizers
• Monitoring with TensorBoard
• Example – GAN on Atari images
• PyTorch Ignite
• Summary

4. The Cross-Entropy Method

• The taxonomy of RL methods
• The cross-entropy method in practice
• The cross-entropy method on CartPole
• The cross-entropy method on FrozenLake
• The theoretical background of the cross-entropy method
• Summary

5. Tabular Learning and the Bellman Equation

• Value, state, and optimality
• The Bellman equation of optimality
• The value of the action
• The value iteration method
• Value iteration in practice
• Q-learning for FrozenLake
• Summary

6. Deep Q-Networks

• Real-life value iteration
• Tabular Q-learning
• Deep Q-learning
• DQN on Pong
• Things to try
• Summary

7. Higher-Level RL Libraries

• Why RL libraries?
• The PTAN library
• The PTAN CartPole solver
• Other RL libraries
• Summary

8. DQN Extensions

• Basic DQN
• N-step DQN
• Double DQN
• Noisy networks
• Prioritized replay buffer
• Dueling DQN
• Categorical DQN
• Combining everything
• Summary
• References

9. Ways to Speed up RL

• Why speed matters
• The baseline
• The computation graph in PyTorch
• Several environments
• Play and train in separate processes
• Tweaking wrappers
• Benchmark summary
• Going hardcore: CuLE
• Summary
• References

10. Stocks Trading Using RL

• Trading
• Data
• Problem statements and key decisions
• The trading environment
• Models
• Training code
• Results
• Things to try
• Summary

11. Policy Gradients – an Alternative

• Values and policy
• The REINFORCE method
• REINFORCE issues
• Policy gradient methods on CartPole
• Policy gradient methods on Pong
• Summary

12. The Actor-Critic Method

• Variance reduction
• CartPole variance
• Actor-critic
• A2C on Pong
• A2C on Pong results
• Tuning hyperparameters
• Summary

13. Asynchronous Advantage Actor-Critic

• Correlation and sample efficiency
• Adding an extra A to A2C
• Multiprocessing in Python
• A3C with data parallelism
• A3C with gradients parallelism
• Summary

14. Training Chatbots with RL

• An overview of chatbots
• Chatbot training
• The deep NLP basics
• Seq2seq training
• Chatbot example
• Dataset exploration
• Training: cross-entropy
• Training: SCST
• Models tested on data
• Telegram bot
• Summary

15. The TextWorld Environment

• Interactive fiction
• The environment
• Baseline DQN
• The command generation model
• Summary

16. Web Navigation

• Web navigation
• OpenAI Universe
• The simple clicking approach
• Human demonstrations
• Adding text descriptions
• Things to try
• Summary

17. Continuous Action Space

• Why a continuous space?
• The A2C method
• Deterministic policy gradients
• Distributional policy gradients
• Things to try
• Summary

18. RL in Robotics

• Robots and robotics
• The first training objective
• The emulator and the model
• DDPG training and results
• Controlling the hardware
• Policy experiments
• Summary

19. Trust Regions – PPO, TRPO, ACKTR, and SAC

• Roboschool
• The A2C baseline
• PPO
• TRPO
• ACKTR
• SAC
• Summary

20. Black-Box Optimization in RL

• Black-box methods
• Evolution strategies
• Genetic algorithms
• Summary
• References

21. Advanced Exploration

• Why exploration is important
• What's wrong with ε-greedy?
• Alternative ways of exploration
• MountainCar experiments
• Atari experiments
• Summary
• References

22. Beyond Model-Free – Imagination

• Model-based methods
• The imagination-augmented agent
• I2A on Atari Breakout
• Experiment results
• Summary
• References

23. AlphaGo Zero

• Board games
• The AlphaGo Zero method
• The Connect 4 bot
• Connect 4 results
• Summary
• References

24. RL in Discrete Optimization

• RL's reputation
• The Rubik's Cube and combinatorial optimization
• Optimality and God's number
• Approaches to cube solving
• The training process
• The model application
• The paper's results
• The code outline
• The experiment results
• Further improvements and experiments
• Summary

25. Multi-agent RL

• Multi-agent RL explained
• The MAgent environment
• Deep Q-network for tigers
• Collaboration by the tigers
• Training both tigers and deer
• The battle between equal actors
• Summary

26. Other Books You May Enjoy

27. Index