![Backpropagation during test time
using the population, rather than mini-batch statistics. Effectively, we process mini-
batches of size 𝑚 and use their statistics to compute:
𝐸 𝑥(𝑘) = 𝐸 𝐵[ 𝜇 𝐵]
𝑉𝑎𝑟 𝑥(𝑘) =
𝑚
𝑚 − 1
𝐸 𝐵[𝜎 𝐵
2 ]
we can use 𝑒𝑥𝑝𝑜𝑛𝑒𝑛𝑡𝑖𝑎𝑙 𝑚𝑜𝑣𝑖𝑛𝑔 𝑎𝑣𝑒𝑟𝑎𝑔𝑒 to estimate the mean and variance to be
used during test time, we estimate the running average of mean and variance as:
𝜇 𝑟𝑢𝑛𝑛𝑖𝑛𝑔 = 𝛼. 𝜇 𝑟𝑢𝑛𝑛𝑖𝑛𝑔 + (1- 𝛼). 𝜇 𝐵
ــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــ
𝜎𝑟𝑢𝑛𝑛𝑖𝑛𝑔
2
= 𝛼. 𝜎𝑟𝑢𝑛𝑛𝑖𝑛𝑔
2
+ (1- 𝛼).𝜎 𝐵
2
where 𝛼 is a constant smoothing factor between 0 and 1 and represents the degree
of dependence on the previous observations .](https://image.slidesharecdn.com/batchnormalizationpresentation-200420214855/85/Batch-normalization-presentation-18-320.jpg)
Batch normalization presentation
- 1. Batch Normalization Accelerating Deep Network Training by Reducing Internal Covariate Shift By : seraj alhamidi Instructor: Associate Prof. Mohammed Alhanjouri June .2019
- 2. About the paper Sergey Ioffe Google Inc., [email protected] Christian Szegedy Google Inc., [email protected] Authors The 32nd International Conference on Machine Learning (2015) presented publishers https://ai.google/research/pubs/pub43442 Journal of Machine Learning Research http://jmlr.org/proceedings/papers/v37/ioffe15.pdf Cornell university https://arxiv.org/abs/1502.03167 paper with over 6000 citations on ICML 2015citations
- 3. Outlines Introduction Issues with Training Deep Neural Networks Batch Normalization Ablation Study Comparison with the State of the art Approaches Some notes My work
- 4. Introduction ILSVRC Competition in 2015 The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) , sponsored by google and Facebook ImageNet, is a dataset of over 15 millions labelled high-resolution images with around 22,000 categories, for classification and localization tasks ILSVRC uses a subset of ImageNet of 1000 categories. On 6 Feb 2015, Microsoft has proposed PReLU-Net which has 4.94% error rate which surpasses the human error rate of 5.1% Five days later, on 11 Feb 2015, Google proposed BN-Inception which has 4.8% error rate. Reach best accuracy in 7% of time need to reach same accuracy
- 5. Issues with Training Deep Neural Networks Vanishing Gradient Saturating nonlinearities (like 𝑡𝑎𝑛ℎ 𝑜𝑟 𝑠𝑖𝑔𝑚𝑜𝑖𝑑) cannot be used for deep networks An example, the sigmoid function and it’s derivative. When the inputs of the sigmoid function becomes larger or smaller , the derivative becomes close to zero. the sigmoid function and its derivativebackpropagation algorithm update rule 𝑤 𝜅 + 1 = 𝑤 𝜅 − 𝛼 𝜕𝐿 𝜕𝑤 𝐿 = 0.5 𝑡 − 𝑎 2 𝑡 ∶ 𝑡𝑎𝑟𝑔𝑒𝑡 , 𝑎 ∶ 𝑜𝑢𝑡𝑝𝑢𝑡 𝑜𝑓 𝑎𝑐𝑡𝑖𝑣𝑎𝑡𝑖𝑜𝑛 𝑎 𝑙 = 𝜎 𝑥 𝑙 𝜎 ∶ 𝑎𝑐𝑡𝑖𝑣𝑎𝑡𝑖𝑜𝑛 𝑧 ∶ 𝑖𝑛𝑝𝑢𝑡 𝑡𝑜 𝑎𝑐𝑡𝑖𝑣𝑎𝑡𝑖𝑜𝑛 𝑥 𝑙 = 𝑤𝑖,𝑗 ∗ 𝑎 𝑙−1 + 𝑤𝑖+1 ,𝑗 ∗ 𝑎 𝑙−1 + ⋯ 𝜕𝐿 𝜕𝑤 = 𝜕𝐿 𝜕𝑎 . 𝜕𝑎 𝜕𝑥 . 𝜕𝑥 𝜕𝑤 ≡ 𝜕𝐿 𝜕𝑎 . 𝜎 𝑥 𝑙 . 𝜕𝑥 𝜕𝑤
- 6. Issues with Training Deep Neural Networks Vanishing Gradient Sigmoid function with restricted inputsRectified linear units 𝑓 𝑥 = 𝑥+ = max(0, 𝑥) Some ways around this are to use: batch normalization layers can also resolve the issue Nonlinearities like Rectified linear units (ReLU) which do not saturate. Smaller learning rates Careful weights initializations
- 7. Issues with Training Deep Neural Networks Internal Covariate shift Covariate – The Features of the Input Data Covariate Shift - The change in the distribution of inputs layers in the middle of a deep neural network, is referred to the technical name “internal covariate shift ”. when the distribution that is fed to the layers of a network should be somewhat: Zero-centered Constant through time and data the distribution of the data being fed to the layers should not vary too much across the mini-batches fed to the network Neural networks learn efficiently
- 8. Issues with Training Deep Neural Networks Internal Covariate shift in deep NN Iteration i Iteration i+1 Iteration i+2 Every time there’s new relation (distribution) specially in deep layers and at the beginning of training
- 9. Issues with Training Deep Neural Networks Take one layer from internal layers Assume has a distribution given below. Also, let us suppose, the function learned by the layer is represented by the dashed line suppose, after the gradient updating, the distribution of x gets changed to something like the loss for this mini-batch is more as compared to the previous loss
- 10. Issues with Training Deep Neural Networks Every time we force the layer (l) to design new Perceptron Because I give it new disruption every time In deep layers , we may have butterfly effect , however the change in 𝒘 at first layers is small, thus make the network unstable
- 11. Batch Normalization The Batch Normalization attempts to normalize a batch of inputs before they are fed to a non-linear activation unit (like ReLU, sigmoid, etc). during training so that the input to the activation function across each training batch has a mean of 0 and a variance of 1 applying batch normalization to the activation σ(Wx + b) would result in σ(BN(Wx + b)) where 𝐵𝑁 is the batch normalizing transform
- 12. Batch Normalization To make each dimension unit gaussian, we apply: 𝑥 𝑘 = 𝑥 𝑘 − 𝐸 𝑥 𝑘 𝑉𝑎𝑟 𝑥 𝑘 where 𝐸 𝑥(𝑘) and 𝑉𝑎𝑟 𝑥(𝑘) are respectively the mean and variance of 𝑘-th feature over a batch. Then we transform 𝑥(𝑘) as: 𝑦 𝑘 = 𝛾 𝑘 𝑥 𝑘 + 𝛽 𝑘 where 𝛾 and 𝛽 are the hyper (learnable) parameters of the so-called batch normalization layer 𝐾 is the 𝑘-th sample in one feature mini batch
- 13. Batch Normalization Transformation of inputs
- 14. Forward Propagation through Batch Normalization layer We have shown the normalization of multiple sample in just one feature Input: Values of 𝐱 over 𝐁 = 𝐱𝟏 … 𝐱 𝐦 a batch; Parameters to be learned 𝛄, 𝛃 Output: 𝓨𝐢 = 𝐁𝐍 𝛄,𝛃(𝐱 𝐢) Flow of computation through Batch Normalization layer 𝝁 𝑩 = 𝟏 𝒎 𝒊=𝟏 𝒎 𝒙𝒊 𝝈 𝑩 𝟐 = 𝟏 𝒎 𝒊=𝟏 𝒎 𝒙𝒊 − 𝝁 𝑩 𝟐 𝒙𝒊= 𝒙𝒊 − 𝝁 𝑩 𝝈 𝑩 𝟐 + 𝝐 𝓨𝒊= 𝜸 𝒙 + 𝜷 = 𝑩𝑵 𝜸,𝜷(𝒙𝒊) 𝜖 𝜖 is a small value 1 ∗ 10−8 for not devided by zero
- 15. Forward Propagation through Batch Normalization layer TWO features
- 16. THE MAGIC Imagine that the network was thought that the optimal that will minimize the cost is to Cancel the BN effect ! Forward Propagation through Batch Normalization layer β = 𝐸 𝑥 = μB 𝛾 = 𝑉𝑎𝑟 𝑥 = 𝜎 𝐵 2 + 𝜖 𝑥𝑖 = 𝑥𝑖 − 𝜇 𝐵 𝜎 𝐵 2 + 𝜖 𝒴𝑖 = 𝛾 𝑥𝑖 + 𝛽 = 𝜎 𝐵 2 + 𝜖 ∗ 𝑥 𝑖 −𝜇 𝐵 𝜎 𝐵 2+𝜖 + 𝜇 𝐵 = 𝑥𝑖 Identity transform 𝛾, 𝛽 Adapted by SGD
- 17. Backpropagation through Batch Normalization layer 𝒴𝑖 = 𝛾 𝑥 + 𝛽 = 𝐵𝑁𝛾,𝛽(𝑥𝑖) 𝝏𝑳 𝝏𝓨𝒊 𝝏𝑳 𝝏 𝒙𝒊 𝜕L 𝜕γi 𝝏𝑳 𝝏𝜷 𝝏𝑳 𝝏𝝈 𝑩 𝟐𝝏𝑳 𝝏𝝁 𝑩 𝝏𝑳 𝝏𝒙 𝒊 SGD 𝜷 𝒌 + 𝟏 = 𝜷 𝜿 − 𝜶 𝝏𝑳 𝝏𝜷 𝜸 𝒌 + 𝟏 = 𝜸 𝜿 − 𝜶 𝝏𝑳 𝝏𝜸 𝝁 𝑩 = 𝟏 𝒎 𝒊=𝟏 𝒎 𝒙𝒊 𝝈 𝑩 𝟐 = 𝟏 𝒎 𝒊=𝟏 𝒎 𝒙𝒊 − 𝝁 𝑩 𝟐 𝒙𝒊= 𝒙𝒊 − 𝝁 𝑩 𝝈 𝑩 𝟐 + 𝝐 𝓨𝒊= 𝜸 𝒙 + 𝜷 = 𝑩𝑵 𝜸,𝜷(𝒙𝒊)
- 18. Backpropagation during test time using the population, rather than mini-batch statistics. Effectively, we process mini- batches of size 𝑚 and use their statistics to compute: 𝐸 𝑥(𝑘) = 𝐸 𝐵[ 𝜇 𝐵] 𝑉𝑎𝑟 𝑥(𝑘) = 𝑚 𝑚 − 1 𝐸 𝐵[𝜎 𝐵 2 ] we can use 𝑒𝑥𝑝𝑜𝑛𝑒𝑛𝑡𝑖𝑎𝑙 𝑚𝑜𝑣𝑖𝑛𝑔 𝑎𝑣𝑒𝑟𝑎𝑔𝑒 to estimate the mean and variance to be used during test time, we estimate the running average of mean and variance as: 𝜇 𝑟𝑢𝑛𝑛𝑖𝑛𝑔 = 𝛼. 𝜇 𝑟𝑢𝑛𝑛𝑖𝑛𝑔 + (1- 𝛼). 𝜇 𝐵 ــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــــ 𝜎𝑟𝑢𝑛𝑛𝑖𝑛𝑔 2 = 𝛼. 𝜎𝑟𝑢𝑛𝑛𝑖𝑛𝑔 2 + (1- 𝛼).𝜎 𝐵 2 where 𝛼 is a constant smoothing factor between 0 and 1 and represents the degree of dependence on the previous observations .
- 19. Ablation Study MNIST dataset 28×28 binary image as input, 3 fully connected (FC) hidden layer with 100 activations each , the last hidden layer followed by 10 activations as there are 10 digits. And the loss is cross entropy loss. BN network is much more stable
- 20. Ablation Study ImageNet of 1000 categories on GoogleNet/Inception(2014) weighing 138GB for the training images, 6.3GB for the validation images, and 13GB for the testing CNN architectures tested
- 21. Comparison with the State of the art Approaches
- 22. Some Notes : cross-entropy loss function We use cross-entropy loss function neural network (1) Computed | targets | correct? ------------------------------------------------ 0.3 0.3 0.4 | 0 0 1 (democrat) | yes 0.3 0.4 0.3 | 0 1 0 (republican) | yes 0.1 0.2 0.7 | 1 0 0 (other) | no neural network (2) Computed | targets | correct? ------------------------------------------------ 0.1 0.2 0.7 | 0 0 1 (democrat) | yes 0.1 0.7 0.2 | 0 1 0 (republican) | yes 0.3 0.4 0.3 | 1 0 0 (other) | no cross-entropy error for the first training −( (ln(0.3) ∗ 0) + (ln(0.3) ∗ 0) + (ln(0.4) ∗ 1) ) = −ln(0.4) average cross-entropy error (ACE) −(ln(0.4) + ln(0.4) + ln(0.1)) / 3 = 1.38 average cross-entropy error (ACE) −(ln(0.7) + ln(0.7) + ln(0.3)) / 3 = 0.64 mean squared error for the first item (0.3 − 0)^2 + (0.3 − 0)^2 + (0.4 − 1)^2 = 0.54 the MSE for the first neural network is (0.54 + 0.54 + 1.34) / 3 = 0.81 The MSE for the second, better, network is (0.14 + 0.14 + 0.74) / 3 = 0.34 (1.38 − 0.64 = 0.74) > (0.81 − 0.34 = 0.47) The 𝑙𝑛() function in cross-entropy takes into account the closeness of a prediction
- 23. Some Notes : Convolutional Neural Network (CNN) It use for Image classification is the task It was developed between 1988 and 1993, at Bell Labs the first convolutional network that could recognize handwritten digits
- 24. Some Notes : Convolutional Neural Network (CNN) Convolution Layer (Conv Layer) Pooling Layer ReLU Layer Fully Connected Layer (Flatten)
- 25. Some Notes : Convolutional Neural Network (CNN)
- 26. Convolution Layer (Conv Layer) convolution works by sliding a window across the input
- 27. Some Notes : Convolutional Neural Network (CNN) 2D filters
- 28. Some Notes : Convolutional Neural Network (CNN) 3D filters
- 29. Pooling Layer (Sub-sampling or Down-sampling) reduce the size of feature maps by using some functions average or the maximum , (hence called down-sampling) make extracted features more robust by making it more invariant to scale and orientation changes.
- 30. ReLU Layer Remove all the black elements from it, keeping only those carrying a positive value (the grey and white colours ) )𝑂𝑢𝑡𝑝𝑢𝑡 = 𝑀𝑎𝑥(𝑧𝑒𝑟𝑜, 𝐼𝑛𝑝𝑢𝑡 to introduce non-linearity in our ConvNet
- 31. Fully Connected Layer (Flatten)
- 32. http://scs.ryerson.ca/~aharley/vis/conv/flat.html
- 33. MY WORK MNIST on google colab Inputs = 28*28 = 784 Layer 1&2 = 100 nodes | Layer 3 = 10 nodes All Activations are sigmoid Cross-entropy loss function The train and test set is already splited in tensorflow the distribution over time of the inputs to the sigmoid function of the first five neurons in the second layer . Batch normalization has a visible and significant effect of removing variance/noise in these inputs.final acc: 99%
- 34. MY WORK caltech dataset 𝑊𝑖𝑡ℎ 𝐵𝑁 #𝑒𝑝𝑜𝑐ℎ = 150 𝐿𝑅 = 1 ∗ 10−3 𝑊𝑖𝑡ℎ𝑜𝑢𝑡 𝐵𝑁 #𝑒𝑝𝑜𝑐ℎ = 150 𝐿𝑅 = 1 ∗ 10−3 𝑤𝑖𝑡ℎ𝑜𝑢𝑡 BN #𝑒𝑝𝑜𝑐ℎ = 250 𝐿𝑅 = 1 ∗ 10−3 final acc: 90.54% final acc: 94.44%final acc: 96.04% We use ten (10) classes from Caltech dataset instead of ImageNet Dataset because it’s huge size classify the input image 1/3 ,1/3 , 1/3 train , validation, test