RL_in_10_min.pptx
- 1. Tech Talk: Reinforcement Learning Tamura Yasuto
- 2. Table of Contents • Theme of This Tech Talk: Stop Saying “Trial and Errors” • Rough Definition of RL (*basic settings) • Planning in Markov Decision Process (MDP) • Interactive Optimization of Policies and Values • Wrapping Up
- 3. Theme of This Tech Talk: Stop Saying “Trial and Errors“ With these charts, you will miss the point in the beginning
- 4. From “Trial and Errors“ to Interactive Value-Policy Updates Agent Environment Action Reward Value Policy This part should be emphasized more
- 5. Table of Contents • Theme of This Tech Talk: Stop Saying “Trial and Errors” • Rough Definition of RL (*basic settings) • Planning in Markov Decision Process (MDP) • Interactive Optimization of Policies and Values • Wrapping Up
- 6. Role of Reinforcement Learnig (RL) in AI Machine learning AI Machine learning Classical models Neural networks Supervised learning Unsupervised learning Reinforcement learning Models How to train
- 7. Rough Definition of RL: Planning Problem • Sequential decision making: optimizing a sequence of actions • Optimizing a “policy”: a “policy” means how to move in a given “state” • Assuming Markov decision processes: next action only depends on the current state Policy Action State Example of planning: navigating a robot
- 8. Table of Contents • Theme of This Tech Talk: Stop Saying “Trial and Errors” • Rough Definition of RL (*basic settings) • Planning in Markov Decision Process (MDP) • Interactive Optimization of Policies and Values • Wrapping Up
- 9. Markov Decision Process (MDP) in Some Expressions Agent Env Action Reward • Typical RL diagram • State transition diagram • Backup diagram (closed) • Graphical model
- 10. MDP: with an Example of Balancing a Bike Or State 0 State 1 State 2 State 3 State 4 Leaning left No move Leaning right
- 11. Plannign in MDP: Some Expressions • Learning how to move optimally in each state No move Lean left Lean right
- 12. Table of Contents • Theme of This Tech Talk: Stop Saying “Trial and Errors” • Rough Definition of RL (*basic settings) • Planning in Markov Decision Process (MDP) • Interactive Optimization of Policies and Values • Wrapping UP
- 13. Values and Policies: with an Example of Balancing a Bike • Value: how good it is to be in a state • Policies: a probability of taking an action in a state State 0: minus reward State 1: low value State 2: high value State 3: low value State 4: minus reward Action 0: Low probability Action 2: High probability Action 1
- 14. Policy updates • Higher probability on actions to the direction of high values State 0: minus reward State 1: low value State 2: high value Action 0: leaning left Action 1: leaning right Then how can a vlaue be learned? Giving higher probability
- 15. Value update: Temporal Difference (TD) Learning • TD learning: updating values by filling a gap between expectation and actual rewards If you lean left, the values is low. As expected! TD loss is low Leaning right would not be good because value is low. I was wrong. There is no bad reward. Let’s update the value. TD loss is high Learning could happen without explicit rewards
- 16. Interactive Updates of Value and Policy Value updates (TD learning) Policy updates
- 17. Table of Contents • Theme of This Tech Talk: Stop Saying “Trial and Errors” • Rough Definition of RL (*basic settings) • Planning in Markov Decision Process (MDP) • Interactive Optimization of Policies and Values • Wrapping Up
- 18. Wrapping Up • RL formulation: a planning problem by optimizing a policy • Simple assumption of MDP : an action only depends on the current state • Importance of a value: updating a policy by evaluating how good to be in • TD learning: updating values by filling a gap between estimations on values and actual rewards