Deep learning from scratch
- 1. Eran Shlomo, IPP tech lead, Haifa [email protected] ©
- 2. About me Haifa IoT Ignition lab and IPP(Intel ingenuity partnership program) tech lead. Intel Perceptual computing. Compute, cloud and embedded expert. Maker and Entrepreneur Focus on Data science and Machine learning in recent years
- 3. Agenda Lets talk some theory Lets define a problem Time to code our network Meet the pro tools Time to fancier netwroks.
- 4. By the end of the day… You will have: • Get some intuition on what is DL, what you can use it for. • Have understanding of the mechanics behind deep learning. • Get basic feeling of the concepts and how does DL works. • Some hands on known tools. • A list of pointers to continue your learning and experimenting. You will not have: • Practical experience on solving problems using DL • Understanding of the different types of networks and their usage • The math skills required to be an expert.
- 5. We are going to work in try, catch up • Along the way we have exercises, you will get time to try them • Usually next slide will contain the solution • So for every task: • Try • Catch up once solution is on board, focus on understanding the solution • Make sure you have it working, each step is required for the one after it.
- 6. Buzzwords alignment attempt AI Machine learning Supervised learning Deep learning Machine reasoning Automated tasks Train based on data Neural networks input logic output input output logic
- 7. Assembly C (compiler) C++(OOP) JAVA(managed) Python (run time) Where we are in technology timeline perspective Model protos High level (keras) ???? ???? ????
- 8. Deep learning – basic anatomy Data driven Training a model Input, output and hidden neurons Input layer Hidden layer(s) Output layer Deep learning Many hidden (deep) layers
- 9. The essence of deeplearning Xi YiWij(1) Wij(2) W11(1) X1 Y1 W11(2) 𝑌 = 𝑓 𝑋 = 𝑊𝑋+b Deep network is essentially a function we train to detect some pattern b (bias) is omitted in this drawing Why is the sudden success ? A lot of data A lot of compute Improved networks
- 10. Before we start , some Math … • As data science is becoming part of every business math gaining extra popularity. • Question you need to ask yourself if you wish to go deeper into the field – Do I Want/Can refresh/increase my math skills. • For our basic deeplearning course we need some: • Algebra, Mainly around Matrix/Vector operations • Calculus, Mainly around derivatives • You can never get enough of statistics in data science, go over variance, mean, distributions, probabilities • Python
- 11. Some math references to get start with https://www.youtube.com/watch?v=K5BLNZw7UeU Matrix operations https://www.youtube.com/watch?v=kuixY2bCc_0 Multiplying matrices https://www.youtube.com/watch?v=rAof9Ld5sOg Derivatives https://www.youtube.com/watch?v=TUJgZ4UDY2g The chain rule https://www.youtube.com/watch?v=ZkjP5RJLQF4 Linear regression https://www.youtube.com/watch?v=_Po-xZJflPM Logistic regression https://www.youtube.com/watch?v=Y4lTTHua0TE Mean, Variance,…
- 12. Lets practice some basics • We are working with python 3.5 . you are encouraged to work with conda package manager but pip is oke as well. • numpy is THE math operations package for python, we will be using it to play with matrices, install it. • Lets create 5 random normal numbers, making sure numpy is good to go • Visualization is very important in general and in the course, install matplotlib • Visualize 100 random numbers like the example above.
- 13. Normal random We got 100 normally distributed numbers, lets create a histogram of them By default our normal distribution is with mean 0 and variance of 1. Create 2 matrices of 10x10: A = N(0,1), B =N(3,16) plot their histogram Hint: Flatten the matrix to histogram Type equation here. X~N(M, σ2 ) Variancemean
- 14. Normal random You can switch between the standard normal to any normal, Z=N(0, 12) Code a function that multiply two matrices explicitly, you can assume inputs are nd array, don’t use numpy matrix multiplication operator. def mul_matrix(a,b): pass Hint: for loop is going over rows in numpy, matrix.T is the transpose
- 15. Matrix multiplication Validate your code using numpy Hint: numpy.dot
- 16. Neural networks – Background and inspiration It is pretty common to compare neural networks to how our brain works: • Coupled well with the term AI • Has some sense in it, as many different researches show. Yet we are a bit long from really understanding how the brain works. 𝑘=0 𝑛 𝑊𝑋 W1 W2 W3 X1 X2 X3 𝑓(𝑥)
- 17. Artificial neural networks Output=f( 𝑘=0 𝑛 𝑊𝑋), where : WX – inputs multiplied by weights F(x) is an activation function Common activation functions: sigmoid, relu, tanh, linear, … 1. Code the sigmoid function Use numpy, z can be a vector def sigmoid(z): …2. Plot 100 points of your sigmoid. Hint: plt.plot(X, Y,'co') draw points only
- 18. Synthetic problem for our neural network One of the biggest challenges with deeplearning is data, and a lot of it. We will override this problem using synthetic problem, we will model predefined function. We usually divide our dataset to (at least) two groups: • Training, ~70% of our data. • Test, 30% of our data. In real life cases you are likely to have validation as well. Our problem is to predict some function behavior given points from this function, later comparing our model into new generated points from the function. Lets example
- 19. Utils.py You got utils.py module, this module contains some help functions. Git repo https://gitlab.com/eshlomo/EazyDnn , utils.py under base_network Lets generate a pattern: • Signature: • Function is given with range, number of samples and generator, which is the function itself. • Generate 100 samples of 𝑓 𝑥 = 𝑥2 between -1 to 1 • Plot the generated function
- 20. Before we go into our network • This is a lot of info to push into short time. • You are likely not be able to follow all details in real time, In order to fully feel you got it you need to train and train and … • We are going through these details to give you a solid base for self learning. • Don’t get too alarmed from the math. • Feel free to contact me
- 21. Lets model a line 𝑓 𝑥 = 𝑥2Previous ex. solution We will create the following network 𝑥 𝑓 𝑥 𝑓 𝑥 = 𝑥 • Generate 100 samples of 𝑓 𝑥 = 𝑥, between 0 to 1 • Plot the generated function 𝑤11 (1) 𝑤12 (1) 𝑤13 (1) 𝑤 1 𝑤 2 𝑤11 (2) 𝑤21 (2)
- 22. Network basics We are going to train our network to estimate our line, in other words: • We are going to find the best vectors w1,w2 that will make our network model our line. • Training is an iterative process in which in every iteration we minimize our model error be small change to our weight vectors. • For that purpose we define a cost (error) function on our model. Previous ex. solution 𝑓 𝑥 = 𝑥
- 23. Cost function • Lets mark our model output as 𝑌, and our real output as 𝑌 • We use Quadratic cost (marked with J), Also known as mean squared error, maximum likelihood, and sum squared error :Err𝑜𝑟 = 𝐽 𝑌 = 1 2 (𝑌 − 𝑌)2 • Keep in mind we know the real output, we have training data==ground truth== annotated data. • Since we want to minimize our error, we would like to move our weights against the derivative at any given iteration. • Similar to finding minimum of function in calculus, only in numeric way, This process is called gradient decent.
- 24. Gradient decent Lets start creating our network, create a class that will manage our network: • 3 layers, their size is constructor parameter. • 2 weight matrices • Init all weights with standard random normal A process in which every iteration we: • Predict our output – Forward pass • Calculate our error using our cost function = prediction – ground truth • Calculate the error derivative in reference for our weights. • Update each weight with small Δ opposite to the gradient direction (minimize) – Backward pass • Δ is called our learning rate
- 25. Our forward pass 𝑘=0 𝑛 𝑊𝑋 W1 W2 W3 X1 X2 X3 𝑠𝑖𝑔𝑚𝑜𝑖𝑑 • Add a method to our class, called forward • This method will calculate 𝑌 our model predicted output • Use the following naming conventions: • 𝑍(𝑛) =𝑊(𝑛−1) 𝑋 • 𝑎(𝑛) =𝑠𝑖𝑔𝑚𝑜𝑖𝑑(𝑍(𝑛) )=𝑓𝑎(𝑍(𝑛) ) • Forward will return our model output, AKA sigmoid of the last layer activation sum - 𝑓𝑎(𝑍(3) ). 𝑍(2) 𝑎(2)
- 26. Lets add our cost function • Add a method to our class, called cost • This method will calculate our cost for every iteration • It gets as parameters our input and output and returns the cost 𝐽 𝑌 = 1 2 (𝑌 − 𝑌)2 • Keep in mind 𝑌=forward(X)
- 27. Back propagation Once we have the cost (error) we want to calculate the derivative of each weight in reference to the cost. Remember – we want to move each weight in the opposite of the direction to its error. We want to minimize 𝐽 𝑌 = 1 2 (𝑌 − 𝑌)2, where 𝑌 = 𝑓(𝑊𝑋) 𝐽 𝑊 = 1 2 (𝑌 − 𝑓(𝑊𝑋))2 So we need to calculate : 𝜕𝐽 𝜕𝑊 𝐽 𝑊 We have a composition of parameters, We need to use the chain rule
- 28. The chain rule • https://en.wikipedia.org/wiki/Chain_rule • A way to compute the derivative of function composition We want to calculate 𝜕𝐽 𝜕𝑊 𝐽 𝑊 , lets do some chaining 𝜕𝐽 𝜕𝑊 𝐽 𝑊 = 𝜕𝐽 𝜕𝑊(1) 𝐽 𝑊 + 𝜕𝐽 𝜕𝑊(2) 𝐽 𝑊 Lets start with 𝜕𝐽 𝜕𝑊(2) 𝐽 𝑊 : 𝜕𝐽 𝜕𝑊(2) 1 2 (𝑌 − 𝑌)2 = 𝑌 − 𝑌 𝜕 𝑌 𝜕𝑊 2 𝑌= 𝑓𝑎( 𝑍(3) ), 𝜕 𝑌 𝜕𝑊 2 = 𝜕 𝑌 𝜕𝑍(3) 𝜕𝑍(3) 𝜕𝑊 2 we need to calculate sigmoid derivative 𝑍(3) = 𝑎(2) 𝑊(2) , 𝜕𝑍(3) 𝜕𝑊 2 = 𝑎(2) Linear propagation of the error per weight.
- 29. Some derivatives Code a function called sigmoidPrime , calculates sigmoid derivative for matix Z Code a function called costPrime, calculates 𝜕𝐽 𝜕𝑊(1) , 𝜕𝐽 𝜕𝑊(2) Sigmoid derivative : 𝑑 𝑑𝑧 1 1+𝑒−𝑧 = 𝑒−𝑧 (1+𝑒−𝑧)2
- 30. Our derivatives functions Utils.py contains methods you should add to your class, add them.
- 31. Time for some training Go over the methods you have just added, can you tell what are they doing ? In utils.py there is a class linear_trainer – what is it doing ? In utils.py there is a method test_line – what is it doing ? Create a network instance, training and test data and train your network using the function test_line.
- 32. Lets run some more Much better, now lets try 100K. mmm… same result Ideas ? Default of training iterations number is 100, lets make it 10000
- 33. When error is too high We usually tend to (in few minutes will get into the why): • Do more training time • Get more data • Get bigger (deeper) model, usually comes with more data. Do the following: • Install scipy • Increase your hidden layer to size 30 • Replace the linear trainer with BFGS_trainer inside the method test_line– find it in utils.py
- 34. Optimization Gradient decent looks for minimum and can suffer from these problems: • Stuck in local minima • Stuck in plateau • Learning rate is too big to reach minima, bouncing… Read more @ http://sebastianruder.com/optimizing- gradient-descent/
- 35. The Bias Variance tradeoff We can look at our model error as follows: noise model error Total Error Our error usually comes from combination of these two, These are all equivalent: • High variance=modeling noise=not enough data=model too big=overfit • High bias =model too simple=underfit
- 36. Bias / Variance 0 20 40 60 80 100 120 140 160 0 5 10 15 Good model 0 20 40 60 80 100 120 140 160 0 5 10 15 High bias 0 20 40 60 80 100 120 140 160 0 5 10 15 High Variance How can you tell which one of those do you have ?
- 37. Rules of thumb regarding Bias/Variance • Good accuracy on training and test Good model • Good accuracy on training, poor on test Overfit • Poor on both underfit Put back our training iterations on 1000, still BGFS Generate the training data as before range 0-1 but test data on range 0-2 How does it look ? Can you guess why ?
- 38. Deep neural nets have limits • On general the network is trained on bounded data • It is likely not to generalize well out of bound • So you need your data sets contain all data range OR • Have more suitable model, for curve prediction(time series) RNN might have been a better choice here. • Try to fit the following curve 2.5𝑒 −𝑥 2 cos(𝜋𝑥) , range 0- 10 • Install tensorflow (cpu version, make sure you are on python 3.5.x) • Install keras • Create our model using Keras(google time…): • Sequential model • Dens layers are the sum • Use relu activation
- 39. Next steps if you wish to get deeper • CNN and caffee: http://adilmoujahid.com/posts/2016/06/introduction-deep- learning-python-caffe/ • Udacity deep learning course (TF examples walkthrough) • Andrew NG ML course on courser • Geof Hinton neural netwroks on coursera • Stanford CS231 on youtube • RNN: http://karpathy.github.io/2015/05/21/rnn-effectiveness/ • Quick round tour on different networks , great youtube series: • https://www.youtube.com/playlist?list=PLjJh1vlSEYgvGod9wWiydumYl8hOXixNu
Editor's Notes
- #4: Smart Home Industry 4.0 Retail Autonomous cars Robotics Medical FinTech Cognitive computing 5G Wearables