![● Build a graph computing a neural net inference.
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('MNIST_data', one_hot=True)
x = tf.placeholder("float", shape=[None, 784])
W = tf.Variable(tf.zeros([784,10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b)
Example TensorFlow fragment](https://image.slidesharecdn.com/k2jeffdean-160609173832/85/Large-Scale-Deep-Learning-with-TensorFlow-23-320.jpg)
![● Automatically add ops to calculate symbolic gradients
of variables w.r.t. loss function.
● Apply these gradients with an optimization algorithm
y_ = tf.placeholder(tf.float32, [None, 10])
cross_entropy = -tf.reduce_sum(y_*tf.log(y))
opt = tf.train.GradientDescentOptimizer(0.01)
train_op = opt.minimize(cross_entropy)
Python API for Machine Learning](https://image.slidesharecdn.com/k2jeffdean-160609173832/85/Large-Scale-Deep-Learning-with-TensorFlow-24-320.jpg)
More Related Content
![[모두의연구소] 쫄지말자딥러닝](https://cdn.slidesharecdn.com/ss_thumbnails/20160528-160529034614-thumbnail.jpg?width=560&fit=bounds)
Similar to Large Scale Deep Learning with TensorFlow (20)
Large Scale Deep Learning with TensorFlow
- 1. Large-Scale Deep Learning With TensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many other people at Google
- 2. We can now store and perform computation on large datasets MapReduce
- 3. We can now store and perform computation on large datasets But what we really want is not just raw data, but computer systems that understand this data MapReduce
- 4. What do I mean by understanding?
- 5. What do I mean by understanding?
- 6. What do I mean by understanding?
- 8. “cat” ● A powerful class of machine learning model ● Modern reincarnation of artificial neural networks ● Collection of simple, trainable mathematical functions What is Deep Learning?
- 9. “cat” ● Loosely based on (what little) we know about the brain What is Deep Learning?
- 10. What is Deep Learning? ● Each neuron is connected to a small subset of other neurons. ● Based on what it sees, it decides what it wants to say. ● Neurons learn to cooperate to accomplish the task. Commonalities with real brains:
- 11. What is Deep Learning? Each neuron implements a relatively simple mathematical function. But the composition of 106 - 109 such functions is surprisingly powerful.
- 12. ConvNets
- 13. Important Property of Neural Networks Results get better with more data + bigger models + more computation (Better algorithms, new insights and improved techniques always help, too!)
- 14. Growing Use of Deep Learning at Google Android Apps drug discovery Gmail Image understanding Maps Natural language understanding Photos Robotics research Speech Translation YouTube … many others ... Across many products/areas: # of directories containing model description files
- 15. Open, standard software for general machine learning Great for Deep Learning in particular First released Nov 2015 Apache 2.0 license http://tensorflow.org/ and https://github.com/tensorflow/tensorflow
- 16. Strong External Adoption GitHub Launch Nov. 2015 GitHub Launch Sep. 2013 GitHub Launch Jan. 2012 GitHub Launch Jan. 2008 50,000+ binary installs in 72 hours, 500,000+ since Nov, 2015 Most forks of any GitHub repo in 2015, despite only being available starting in Nov, 2015 (source: http://donnemartin.com/viz/pages/2015)
- 17. Motivations ● DistBelief (our 1st system) was scalable and good for production deployment, but not as flexible as we wanted for research purposes ● Better understanding of problem space allowed us to make some dramatic simplifications ● Define a standard way of expressing machine learning ideas and computations ● Short circuit the MapReduce/Hadoop inefficiency
- 20. TensorFlow: Expressing High-Level ML Computations ● Core in C++ ○ Very low overhead Core TensorFlow Execution System CPU GPU Android iOS ...
- 21. TensorFlow: Expressing High-Level ML Computations ● Core in C++ ○ Very low overhead ● Different front ends for specifying/driving the computation ○ Python and C++ today, easy to add more Core TensorFlow Execution System CPU GPU Android iOS ...
- 22. TensorFlow: Expressing High-Level ML Computations ● Core in C++ ○ Very low overhead ● Different front ends for specifying/driving the computation ○ Python and C++ today, easy to add more Core TensorFlow Execution System CPU GPU Android iOS ... C++ front end Python front end ...
- 23. ● Build a graph computing a neural net inference. import tensorflow as tf from tensorflow.examples.tutorials.mnist import input_data mnist = input_data.read_data_sets('MNIST_data', one_hot=True) x = tf.placeholder("float", shape=[None, 784]) W = tf.Variable(tf.zeros([784,10])) b = tf.Variable(tf.zeros([10])) y = tf.nn.softmax(tf.matmul(x, W) + b) Example TensorFlow fragment
- 24. ● Automatically add ops to calculate symbolic gradients of variables w.r.t. loss function. ● Apply these gradients with an optimization algorithm y_ = tf.placeholder(tf.float32, [None, 10]) cross_entropy = -tf.reduce_sum(y_*tf.log(y)) opt = tf.train.GradientDescentOptimizer(0.01) train_op = opt.minimize(cross_entropy) Python API for Machine Learning
- 25. ● Launch the graph and run the training ops in a loop init = tf.initialize_all_variables() sess = tf.Session() sess.run(init) for i in range(1000): batch_xs, batch_ys = mnist.train.next_batch(100) sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys}) Python API for Machine Learning
- 26. MatMul Add Relu biases weights examples labels Xent Graph of Nodes, also called Operations or ops. Computation is a dataflow graph
- 27. with tensors MatMul Add Relu biases weights examples labels Xent Edges are N-dimensional arrays: Tensors Computation is a dataflow graph
- 28. with state Add Mul biases ... learning rate −=... 'Biases' is a variable −= updates biasesSome ops compute gradients Computation is a dataflow graph
- 29. GPU 0 CPU Add Mul biases learning rate Assign Sub ... ... distributedComputation is a dataflow graph
- 30. GPU 0 CPU Add Mul biases learning rate Assign Sub ... ... Assign Devices to Ops ● TensorFlow inserts Send/Recv Ops to transport tensors across devices ● Recv ops pull data from Send ops Send Recv
- 31. GPU 0 CPU Add Mul biases learning rate Assign Sub ... ... Assign Devices to Ops ● TensorFlow inserts Send/Recv Ops to transport tensors across devices ● Recv ops pull data from Send ops Send Recv Send Recv Send Recv Send Recv
- 32. phones single machines (CPU and/or GPUs) … distributed systems of 100s of machines and/or GPU cards Automatically Runs on Variety of Platforms custom ML hardware
- 33. What are some ways that deep learning is having a significant impact at Google?
- 34. “How cold is it outside?” Deep Recurrent Neural Network Acoustic Input Text Output Reduced word errors by more than 30% Speech Recognition Google Research Blog - August 2012, August 2015
- 35. “ocean” Deep Convolutional Neural Network Your Photo Automatic Tag Search personal photos without tags. Google Photos Search Google Research Blog - June 2013
- 37. Google Photos Search “Wow. The new Google photo search is a bit insane. I didn’t tag those… :)”
- 38. We have tons of vision problems Image search, StreetView, Satellite Imagery, Translation, Robotics, Self-driving Cars, www.google.com/sunroof
- 39. “Seeing” Go
- 40. April 1, 2009: April Fool’s Day joke Nov 5, 2015: Launched Real Product Feb 1, 2016: >10% of mobile Inbox replies Smart Reply
- 41. Small Feed- Forward Neural Network Incoming Email Activate Smart Reply? yes/no Smart Reply Google Research Blog - Nov 2015
- 42. Small Feed- Forward Neural Network Incoming Email Activate Smart Reply? Deep Recurrent Neural Network Generated Replies yes/no Smart Reply Google Research Blog - Nov 2015
- 43. Combined Vision + Translation
- 44. Model: A close up of a child holding a stuffed animal. Human: A young girl asleep on the sofa cuddling a stuffed bear. Model: A baby is asleep next to a teddy bear. Image Captions Research
- 46. Experiment Turnaround Time and Research Productivity ● Minutes, Hours: ○ Interactive research! Instant gratification! ● 1-4 days ○ Tolerable ○ Interactivity replaced by running many experiments in parallel ● 1-4 weeks ○ High value experiments only ○ Progress stalls ● >1 month ○ Don’t even try
- 48. Image Model Training Time Hours 10 GPUs 50 GPUs 1 GPU
- 49. Hours 2.6 hours vs. 79.3 hours (30.5X) 10 GPUs 50 GPUs 1 GPU Image Model Training Time
- 50. How Can You Get Started with Machine Learning? Four ways, with varying complexity: (1) Use a Cloud-based API (Vision, Speech, etc.) (2) Run your own pretrained model (3) Use an existing model architecture, and retrain it or fine tune on your dataset (4) Develop your own machine learning models for new problems More flexible, but more effort required
- 51. (1) Use Cloud-based APIs cloud.google.com/translate cloud.google.com/speech cloud.google.com/vision cloud.google.com/text
- 52. (1) Use Cloud-based APIs cloud.google.com/translate cloud.google.com/speech cloud.google.com/vision cloud.google.com/text
- 53. Google Cloud Vision API https://cloud.google.com/vision/
- 54. (2) Using a Pre-trained Image Model yourself with TensorFlow www.tensorflow.org/tutorials/image_recognition/index.html
- 55. For training your own models (3 & 4), two choices: cloud.google.com/ml or Run open-source release on your own physical machines or virtual machines in a cloud hosting environment
- 56. (3) Training a Model on Your Own Image Data www.tensorflow.org/versions/master/how_tos/image_retraining/index.html
- 57. (4) Develop your own machine learning models https://www.tensorflow.org/versions/master/get_started/basic_usage.html
- 58. Example: Combining Vision with Robotics “Deep Learning for Robots: Learning from Large-Scale Interaction”, Google Research Blog, March, 2016 “Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection”, Sergey Levine, Peter Pastor, Alex Krizhevsky, & Deirdre Quillen, Arxiv, arxiv.org/abs/1603.02199 https://www.youtube.com/watch?v=iaF43Ze1oeI
- 59. What Does the Future Hold? Deep learning usage will continue to grow and accelerate: ● Across more and more fields and problems: ○ robotics, self-driving vehicles, ... ○ health care ○ video understanding ○ dialogue systems ○ personal assistance ○ ...
- 60. Conclusions Deep neural networks are making significant strides in understanding: In speech, vision, language, search, … If you’re not considering how to use deep neural nets to solve your vision or understanding problems, you almost certainly should be
- 61. Further Reading ● Dean, et al., Large Scale Distributed Deep Networks, NIPS 2012, research.google. com/archive/large_deep_networks_nips2012.html. ● TensorFlow white paper, tensorflow.org/whitepaper2015.pdf (clickable links in bibliography) ● TensorFlow: A System for Large-Scale Machine Learning, http://arxiv.org/abs/1605.08695 tensorflow.org research.google.com/people/jeff We’re Hiring! See g.co/brain Questions?