Tensor Abuse - how to reuse machine learning frameworks
- 1. © 2017 MapR TechnologiesMapR Confidential 1 Tensor Abuse In the Workplace
- 2. © 2017 MapR TechnologiesMapR Confidential 2 Agenda • What are tensors? • How modern tensor systems work • It's not just machine learning • It’s also not magic • Machine learning as optimization • Animation of optimization algorithm
- 3. © 2017 MapR TechnologiesMapR Confidential 3 Spoilers • Trigger warning: Greek letters ahead – AND lots of animations for developing intuitions • Tensors have a history in physics • We don’t use them that way in machine learning • Tensors are cool because they make it easier to code numerical algorithms • But auto-derivatives are as big a deal • And we can abuse this machinery
- 4. © 2017 MapR TechnologiesMapR Confidential 4 Linear Operators on To the N-th Degree • Tensors were originally invented for differential geometry • That gave us relativity
- 5. © 2017 MapR TechnologiesMapR Confidential 5 But This Isn’t Physics • In computing, we use some of the same words • But they don’t mean the same thing • We don’t need the Greek letters • The point is that we have important patterns of computation – We mostly don’t care about changing coordinate systems
- 6. © 2017 MapR TechnologiesMapR Confidential 6 Basic Operations • Element-wise operations • Outer products • Reductions • Matrix and vector products are special cases
- 7. © 2017 MapR TechnologiesMapR Confidential 7 This is news?!?
- 8. © 2017 MapR TechnologiesMapR Confidential 8 This is news?!? Sounds like APL in sheep’s clothing
- 9. © 2017 MapR TechnologiesMapR Confidential 9 But it really is important because loop structuring is critical
- 10. © 2017 MapR TechnologiesMapR Confidential 10 Why This Matters for Machine Learning • Sums of products and element-wise evaluations are ubiquitous in machine learning • And this is often surrounded by an outer loop • This is susceptible to pipelining in GPU’s, but only if you can see the large-scale patterns in the code
- 11. © 2017 MapR TechnologiesMapR Confidential 11 But, tensors are only half of the story
- 12. © 2017 MapR TechnologiesMapR Confidential 12 Machine Learning Has Changed • Machine learning used to be really hard – Numeric performance was zilch – Training data was poor and small – Learning many layers of a MLP was impossible – Productivity for new approaches hampered by code complexity • Recent advances have changed things (a lot) – Important new regularization techniques – Per coefficient learning rate techniques – New gradient based optimization algorithms – Automated differentiation
- 13. © 2017 MapR TechnologiesMapR Confidential 13
- 14. © 2017 MapR TechnologiesMapR Confidential 14
- 15. © 2017 MapR TechnologiesMapR Confidential 15
- 16. © 2017 MapR TechnologiesMapR Confidential 17 Gradients are a Big Deal • In low dimensions, simple search techniques work well – Line search, evolutionary processes, polyhedron warping all work • It’s different in high dimensions – Need some guide about which direction to go – Too many ways to get sideways
- 17. © 2017 MapR TechnologiesMapR Confidential 18
- 18. © 2017 MapR TechnologiesMapR Confidential 19
- 19. © 2017 MapR TechnologiesMapR Confidential 20 Yeah, But … • Gradients are also a pain in the *** – Traditionally you needed to derive them by hand – For some problems like robot kinematics or astrodynamics, equations would cover many pages • The big change in recent years is automatic differentiation – Some limits on how problems are formulated – But pretty complicated forms can be allowed easily
- 20. © 2017 MapR TechnologiesMapR Confidential 21
- 21. © 2017 MapR TechnologiesMapR Confidential 22
- 22. © 2017 MapR TechnologiesMapR Confidential 23 hidden = tf.nn.elu(x - biases) y_pred = tf.matmul(hidden, weights) loss = tf.reduce_mean((y - y_pred) ** 2)
- 23. © 2017 MapR TechnologiesMapR Confidential 24 The Lessons to Learn • These new systems are astounding – Tensors allow us to encode very complex algorithms – Auto-differentiation allows us to use gradients – New optimizers solve very nasty problems – The same code can drive CPU, GPU or clusters – The code is strange and abstract, but not that hard • Not just for breakfast any more – Machine learning isn’t the only thing you can do with these systems – Go for it!
- 24. © 2017 MapR TechnologiesMapR Confidential 25 Q&A @mapr maprtechnologies [email protected] ENGAGE WITH US