![Initialization
• For bias
• Initialize all to 0
• For weights
• Can’t initialize all weights to the same value
• we can show that all hidden units in a layer will always behave the same
• need to break symmetry
• Recipe: U[-b, b]
• the idea is to sample around 0 but break symmetry](https://image.slidesharecdn.com/neuralnetworks-240415033525-eca63cb5/85/Neural_Networks_scalability_consntency-ppt-24-320.jpg)
Neural_Networks_scalability_consntency.ppt
- 2. Neural Networks • Origins: Algorithms that try to mimic the brain. What is this?
- 3. A single neuron in the brain Input Output
- 4. An artificial neuron: Logistic unit “Bias unit” “Input” “Output” “Weights” “Parameters”
- 5. Visualization of weights, bias, activation function bias b only change the position of the hyperplane range determined by g(.)
- 6. Activation - sigmoid • Squashes the neuron’s pre- activation between 0 and 1 • Always positive • Bounded • Strictly increasing
- 7. Activation - hyperbolic tangent (tanh) • Squashes the neuron’s pre- activation between -1 and 1 • Can be positive or negative • Bounded • Strictly increasing
- 8. Activation - rectified linear(relu) • Bounded below by 0 • always non-negative • Not upper bounded • Tends to give neurons with sparse activities
- 10. Universal approximation theorem ‘‘a single hidden layer neural network with a linear output unit can approximate any continuous function arbitrarily well, given enough hidden units’’ Hornik, 1991
- 11. Neural network – Multilayer Layer 1 “Output” Layer 2 (hidden) Layer 3
- 12. Neural network
- 13. Neural network “Pre-activation” Why do we need g(.)?
- 15. Flow graph - Forward propagation X How do we evaluate our prediction?
- 16. Cost function Logistic regression: Neural network:
- 17. Gradient computation Need to compute:
- 21. Activation - sigmoid • Partial derivative
- 22. Activation - hyperbolic tangent (tanh) • Partial derivative
- 23. Activation - rectified linear(relu) • Partial derivative
- 24. Initialization • For bias • Initialize all to 0 • For weights • Can’t initialize all weights to the same value • we can show that all hidden units in a layer will always behave the same • need to break symmetry • Recipe: U[-b, b] • the idea is to sample around 0 but break symmetry
- 25. Putting it together Pick a network architecture • No. of input units: Dimension of features • No. output units: Number of classes • Reasonable default: 1 hidden layer, or if >1 hidden layer, have same no. of hidden units in every layer (usually the more the better) • Grid search
- 26. Putting it together Early stopping • Use a validation set performance to select the best configuration • To select the number of epochs, stop training when validation set error increases
- 27. Other tricks of the trade • Normalizing your (real-valued) data • Decaying the learning rate • as we get closer to the optimum, makes sense to take smaller update steps • mini-batch • can give a more accurate estimate of the risk gradient • Momentum • can use an exponential average of previous gradients
- 28. Dropout • Idea: «cripple» neural network by removing hidden units • each hidden unit is set to 0 with probability 0.5 • hidden units cannot co-adapt to other units • hidden units must be more generally useful