強化学習の分散アーキテクチャ変遷
27 likes•15,478 views
The document discusses the evolution of distributed architectures in deep reinforcement learning, highlighting various methods such as A3C, GA3C, and IMPALA. It emphasizes accelerated techniques for efficiency in training and performance metrics, as well as comparative studies of different algorithms and their applications in various environments. Key findings address efficiency optimizations, GPU utilization, and the impact of different architectures on learning speed and stability.
![• Include
•
• Gorila, A3C, GA3C, batched A2C, Ape-X, IMPALA
•
• Accelerated Methods for Deep Reinforcement Learning
• Exclude
•
•
• Trace [8]
• Importance Sampling, Tree-backup, Retrace
• UNREAL [9]
•
•](https://image.slidesharecdn.com/reinforcementlearningarchitecture14-180313123842/85/-2-320.jpg)
![DistBelief[10]
A3C
GA3C
batchedA2C
IMPALA
Gorila
Ape-X
AcceleratedMethods
2012 2015 2016 2017 2018](https://image.slidesharecdn.com/reinforcementlearningarchitecture14-180313123842/85/-3-320.jpg)
![Gorila [1]](https://image.slidesharecdn.com/reinforcementlearningarchitecture14-180313123842/85/-5-320.jpg)
![A3C [2]](https://image.slidesharecdn.com/reinforcementlearningarchitecture14-180313123842/85/-9-320.jpg)
![A3C
[3] GA3C
:
•
• on-policy
•
• 1 CPU (GPU )](https://image.slidesharecdn.com/reinforcementlearningarchitecture14-180313123842/85/-10-320.jpg)
![GA3C [3]](https://image.slidesharecdn.com/reinforcementlearningarchitecture14-180313123842/85/-13-320.jpg)
![batched A2C [4]](https://image.slidesharecdn.com/reinforcementlearningarchitecture14-180313123842/85/-28-320.jpg)
![The number of actors
Learning rate Actor ( ) 0.0007 . [13]
. .](https://image.slidesharecdn.com/reinforcementlearningarchitecture14-180313123842/85/-34-320.jpg)
![Ape-X [5]](https://image.slidesharecdn.com/reinforcementlearningarchitecture14-180313123842/85/-37-320.jpg)
![IMPALA [6]](https://image.slidesharecdn.com/reinforcementlearningarchitecture14-180313123842/85/-46-320.jpg)
![Efficiency Optimisations
• GPU CPU
• A3C IMPALA
• GA3C, A2C, Ape-X
• TensorFlow
•
• XLA [11]
• cuDNN [12]](https://image.slidesharecdn.com/reinforcementlearningarchitecture14-180313123842/85/-49-320.jpg)
![Retrace
• Retrace[8] ”trace cutting”
• は時刻tのTDが前回の時刻sの価値関数の更新にど
れだけ影響を与えるか測定する
• πとμが⼤きく異なれば(オフポリシー)、よりバリアンス(学
習の変動)が⼤きくなる
• はバリアンスの削減係数
• このテクニックを⽤いても収束する値に変化はない](https://image.slidesharecdn.com/reinforcementlearningarchitecture14-180313123842/85/-53-320.jpg)
![Accelerated Methods
for Deep Learning [7]](https://image.slidesharecdn.com/reinforcementlearningarchitecture14-180313123842/85/-61-320.jpg)
![• Pong 4 (A3C 4 )
•
•
•
•
• GPU
•
• ,
•
• / Retrace [8]
• (UNREAL) [9]](https://image.slidesharecdn.com/reinforcementlearningarchitecture14-180313123842/85/-80-320.jpg)
強化学習の分散アーキテクチャ変遷
- 2. • Include • • Gorila, A3C, GA3C, batched A2C, Ape-X, IMPALA • • Accelerated Methods for Deep Reinforcement Learning • Exclude • • • Trace [8] • Importance Sampling, Tree-backup, Retrace • UNREAL [9] • •
- 3. DistBelief[10] A3C GA3C batchedA2C IMPALA Gorila Ape-X AcceleratedMethods 2012 2015 2016 2017 2018
- 4. • Gorila • A3C • GA3C • batched A2C • Ape-X • IMPALA • Accelerated Methods for Deep Reinforcement Learning
- 5. Gorila [1]
- 6. DQN ( )
- 7. Gorila
- 8. Results DQN (6 19 ) DQN 12~14 (6 ) DQN 41/49
- 9. A3C [2]
- 10. A3C [3] GA3C : • • on-policy • • 1 CPU (GPU )
- 11. A3C • DNN • CNN2 +FC +Softmax ×2(policy, value) • • • •
- 12. Experiments LSTM . • • Multi-Step A3C Pong 4
- 13. GA3C [3]
- 14. GA3C • A3C CPU/GPU • DQN, Double-DQN, Dueling Double DQN GPU • SoTa GPU A3C • AlphaGO • 50GPU • Gorilla DQN 31 100 • GPU • Queue • • TensorFlow • SoTa
- 15. A3C • 16 , 16CPU • 4 (Atari ,Brockman) • • • • GPU • A3C Replay Memory
- 16. Hybrid CPU/GPU A3C (GA3C)
- 17. Performance metrics and trade-offs • • CPU-GPU • GPU • GPU • GA3C • Predictor <=> GPU • Trainer <=> GPU • <=>
- 18. • Training Per Second (TPS) • • Predictions Per Second (PPS) • • A3C 5 … • PPS TPS × 5
- 19. Dynamic adjustment of trade- offs • TPS • , DNN , • Atari BOXING Atari PONG
- 20. Policy lag in GA3C • A3C GA3C • • k
- 21. Maximizing training speed GPU : A3C
- 22. GPU utilization and DNN size Large DNN TPS 7% … GPU ( 12%UP) : A3C
- 23. Effect of TPS on learning speed
- 24. Compares scores A3C 4 ( ) GA3C 1
- 26. Policy lag, learning stability and convergence speed • • TPS • • TPS, , 1~40 …
- 27. GA3C • CPU/GPU • • GA3C A3C • ( )
- 28. batched A2C [4]
- 29. batched A2C • Gorila • DQN • Actor Learner • A3C • • GA3C • GPU • • • • Gorilla Actor, A3C GA3C , • • Gorila A3C 1 • GA3C ( , ) • Atari SoTa
- 30. Parallel Framework for Deep Reinforcement Learning A3C on-line experience memory
- 31. Algorithm
- 32. Experiments • Atari 12 • Python TensorFlow • • 4 (Intel i7-4790K) • Nvidia 980 Ti GPU • • Gorila, A3C, GA3C • Arch(nips): Conv×2 + FC×2 • Arch(nature): Conv×3 + FC×2 (nips )
- 33. Results
- 34. The number of actors Learning rate Actor ( ) 0.0007 . [13] . .
- 35. Time usage in the game of Pong nips (ne=32) nature 22% , CPU 41% : GPU
- 36. batched A2C • GPU • , • •
- 37. Ape-X [5]
- 38. Ape-X • • • Actor • • Experience Replay Memory • Prioritized Experience Replay
- 40. Distributed Prioritized Experience Replay • Gorila • Replay Memory • Actor • Prioritized DQN … • 1 • Actor • Ape-X Actor
- 41. Experiments (Ape-X DQN) • Atari 57 • 12.5K FPS =360Actor×139FPS/4Repeat • Actor 100 • 19 /sec, 16, 512 • png • Actor 400 Learner • Actor ε ( ) ε-greedy • Experience Replay 20 • 100 FIFO • Priority Exponent = 0.6, Importance Sampling Exponent = 0.4
- 43. Scaling the number of actors (Ape-X DQN) Actor
- 44. Varying the capacity of the replay
- 45. Conclusion • Prioritized replay • , SoTa • • •
- 46. IMPALA [6]
- 47. IMPALA • • • A2C GA3C • IMPALA • V-trace • • DMLab-30 • Atari57 •
- 48. IMPALA Actor n … 1. Learner 2. , , LSTM Learner Learner Actor Learner π Actor μ ( ) V- trace GPU Point: (Actor) (Learner) A3C . ( )
- 49. Efficiency Optimisations • GPU CPU • A3C IMPALA • GA3C, A2C, Ape-X • TensorFlow • • XLA [11] • cuDNN [12]
- 50. V-trace • Actor Learner • Actor-Learner • V-trace Actor-Critic Notation (MDP) : : : (Policy μ): μ π
- 51. V-trace target • • Temporal Difference • Importance Sampling • (π=μ) => n-step
- 52. Importance Sampling • TD • • μ π • 0 behavior policy • target policy
- 53. Retrace • Retrace[8] ”trace cutting” • は時刻tのTDが前回の時刻sの価値関数の更新にど れだけ影響を与えるか測定する • πとμが⼤きく異なれば(オフポリシー)、よりバリアンス(学 習の変動)が⼤きくなる • はバリアンスの削減係数 • このテクニックを⽤いても収束する値に変化はない
- 54. Experiments • • • • DeepMind Lab 30 , Atari 57 •
- 55. Computational Performance • A3C, Batched A2C (Shallow Model)
- 56. Single-Task Training • 5 DeepMind Lab • planning task • maze navigation task × 2 • laser tag task • simple fruit collection task •
- 57. Convergence and Stability • 2/5 IMPALA ( ) • V-trace • • IMPALA A3C ( )
- 58. V-trace Analysis • No-correction: No off-policy • ε-correction • logπが⼩さくなりすぎないように 勾配計算時に微⼩εを加算 • 1-step importance sampling • 各ステップで重みをかける • V-traceのtraceない版 • V-trace V-trace Replay Replay
- 59. Multi-Task (Atari) • Atari 57 • IMPALA A3C shallow 1 IMPALA shallow, deep A3C shallow
- 60. IMPALA • IMPALA • • オフポリシーアルゴリズムV-trace • 他のオフポリシーActor-Critic⼿法と⽐較して安定 • 実験 • DMLab-30とAtari57でのマルチタスク学習 • A3C⽐較して優れたパフォマーンス
- 61. Accelerated Methods for Deep Learning [7]
- 62. Accelerated Methods • • CPU GPU • • • • • Atari • : NVIDIA DGX-1 • 40 CPU cores, 8P 100GPU
- 63. Related Work • Gorila • sub-linear • Ape-X • prioritized replay • CPU GPU • A3C • • GA3C • CPU A3C GPU • • IMPALA • GPU • V-trace • PAAC (batched A2C) • batched A2C
- 64. Parallel, Accelerated RL Framework • • … • CPU • Deep Neural Network
- 65. Synchronized Sampling • CPU , • • CPU • CPU • サンプリングと推論が交互だが、2グループに分けて処理を進める⼯夫もできる.
- 66. Synchronous Multi-GPU Optimization • GPU • • • GPU Reduce ( ) • • NVIDIA Collective Communication Library GPU
- 67. Asynchronous Multi-GPU Optimization • GPU Sampler Learner • 1. GPU 2. GPU 3. • CPU • 2,3 4. GPU
- 69. Atari Frame Processing • Minh et al., 2015(Human Level…) • • 2 • 2 104×80 • • Q 3 (DQN-Net) • 2 (A3C-Net)
- 72. PPO • PPO (8 ×256=2048) • 環境1つあたりのバッチサイズを256から4に減らしていった 並列性を⾼くするとよくなったゲームもあれば悪くなったものもあった
- 73. Q-Value Learning with Large Training Batches • DQN • 32 2048 . 512 . • 32 • • (2.5, 7.5, 15*10^4) • Categorical DQN • DQN 2048 •
- 74. Update Rule • Adam RMSProp • Categorical DQN, Rainbow RMSProp Adam • Learner • RMSProp Adam
- 75. Learning Speed • A2C,A3C,PPO,APPO • (10 ) • PPO Pong 4 • 256 A2C 25,000 /sec (=90million/hour)
- 77. Accelerated Methods • RL • Q • … • Atari • … • Atari •
- 79. • Gorila • DQN • A3C • (Actor-Critic) • GA3C • ( ) GPU • batched A2C • • Ape-X • Prioritized Replay • IMPALA • Importance Sampling(Retrace) • Accelerated Methods for Deep Reinforcement Learning •
- 80. • Pong 4 (A3C 4 ) • • • • • GPU • • , • • / Retrace [8] • (UNREAL) [9]
- 82. 1. Nair, Arun, et al. “Massively parallel methods for deep reinforcement learning”. arXiv preprint arXiv: 1507.04296, 2015. 2. Mnih, Volodymyr, et al. "Asynchronous methods for deep reinforcement learning". Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, 2016. 3. Babaeizadeh, Mohammad, et al. "GA3C: GPU-based A3C for deep reinforcement learning". NIPS Workshop, 2016. 4. Clemente, Alfredo V., et al. "Efficient parallel methods for deep reinforcement learning". CoRR, abs/ 1705.04862, 2017. 5. Horgan, D., et al. “Distributed Prioritized Experience Replay”. ArXiv e-prints, March 2018. 6. Espeholt, Lasse, et al. "IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures”. arXiv preprint arXiv: 1802.01561, 2018 7. Stooke, Adam, Abbeel, Pieter. “Accelerated Methods for Deep Learning”. arXiv preprint arXiv: 1803.02811, 2018
- 83. 8. Munos, Re ́mi, et al. “Safe and efficient off-policy reinforcement learning”. In Advances in Neural Information Processing Systems, pp. 1046–1054, 2016. 9. Jaderberg, Max, et al. “Reinforcement learning with unsupervised auxiliary tasks”. International Conference on Learning Representations, 2017. 10.Dean, Jeffrey, et al. “Large scale distributed deep networks”. In Advances in Neural Information Processing Systems 25, pp. 1223–1231, 2012. 11.TensorFlow w/XLA: TensorFlow, Compiled! • https://autodiff-workshop.github.io/slides/JeffDean.pdf 12.Chetlur, Sharan, et al. cudnn: Efficient primitives for deep learning. CoRR, abs/1410.0759, 2014. 13.Goyal, Priya, et al. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017. 14.Intuitive RL: Intro to Advantage-Actor-Critic (A2C) • https://hackernoon.com/intuitive-rl-intro-to-advantage-actor-critic-a2c-4ff545978752