CMU 11-785 L07 Optimizers and regularizers

Optimizers

• Momentum and Nestorov’s method improve convergence by normalizing the mean (first moment) of the derivatives
• Considering the second moments
  • RMS Prop / Adagrad / AdaDelta / ADAM¹
• Simple gradient and momentum methods still demonstrate oscillatory behavior in some directions²
  • Depends on magic step size parameters (learning rate)
• Need to dampen step size in directions with high motion
  • Second order term (use variation to smooth it)
  • Scale down updates with large mean squared derivatives
  • scale up updates with small mean squared derivatives

RMS Prop

• Notion
  • The squared derivative is ${\partial }_w^{2}D={\left({\partial }_{w}D\right)}^2$
  • The mean squared derivative is $E\left[{\partial }_W^{2}D\right]$
• This is a variant on the basic mini-batch SGD algorithm
  • Updates are by parameter

$$E{\left[{\partial }_w^{2}D\right]}_k=\gamma E{\left[{\partial }_w^{2}D\right]}_{k-1}+(1-\gamma ){\left({\partial }_w^{2}D\right)}_k$$

$$w_{k+1}=w_k-\frac{\eta }{\sqrt{E{\left[{\partial }_w^{2}D\right]}_k+ϵ}}{\partial }_{w}D$$

• If using the same step over a long period, $\sqrt{E{\left[{\partial }_w^{2}D\right]}_k+ϵ}\approx |{\partial }_{w}D|$
  • So $w_{k+1}=w_k-\text{sign}({\partial }_{w}D)\eta$
  • Only the sign remain, similar to RProp

Adam

• RMS prop only considers a second-moment normalized version of the current gradient
• ADAM utilizes a smoothed version of the momentum-augmented gradient
  • Considers both first and second moments

$$m_k=\delta m_{k-1}+(1-\delta ){\left({\partial }_{w}D\right)}_k$$

$$v_k=\gamma v_{k-1}+(1-\gamma ){\left({\partial }_w^{2}D\right)}_k$$

$$\widehat{m_k}=\frac{m_k}{1-{\delta }^k},{\hat{v}}_k=\frac{v_k}{1-{\gamma }^k}$$

$$w_{k+1}=w_k-\frac{\eta }{\sqrt{{\hat{v}}_k+ϵ}}{\hat{m}}_k$$

• Typically $\delta \approx 1$, initalize $m_{k-1}, v_{k-1} \approx 0 $, so $1-\delta \approx 0$, will be very slow to update in the beginning
  • So we need $\widehat{m_k}=\frac{m_k}{1-{\delta }^k}$ term to scale up in the beginning

Tricks

• To make the network converge better, we can consider the following aspects
  • The Divergence
  • Dropout
  • Batch normalization
  • Gradient clipping
  • Data augmentation

Divergence

• What shape do we want the divergence function would be?
  • Must be smooth and not have many poor local optima
  • The best type of divergence is steep far from the optimum, but shallow at the optimum
    • But not too shallow(hard to converge to minimum)
• The choice of divergence affects both the learned network and results

• Common choices

  • L2 divergence

  $$Div=\frac{1}{2}\sum _i{\left(y_i-d_i\right)}^2$$

  • KL divergence

  $$Div=\sum _i{d}_i\log \left(d_i\right)-\sum _i{d}_i\log \left(y_i\right)$$

• L2 is particularly appropriate when attempting to perform regression

  • Numeric prediction
  • For L2 divergence the derivative w.r.t. the pre-activation of the output layer is :
    • ${\nabla }_z\frac{1}{2}\Vert y-d{\Vert }^2=(y-d)J_y(z)$
  • We literally “propagate” the error $(y-d)$ backward
  • Which is why the method is sometimes called “error backpropagation”

• The KL divergence is better when the intent is classification

  • The output is a probability vector

Batch normalization

• Covariate shifts problem

  • Training assumes the training data are all similarly distributed (So as mini-batch)
  • In practice, each minibatch may have a different distribution
  • Which may occur in each layer of the network
  • Minimize one batch cannot give the correction of other batches

• Solution

  • Move all batches to have a mean of 0 and unit standard deviation
  • Eliminates covariate shift between batches

• Batch normalization is a covariate adjustment unit that happens after the weighted addition of inputs (affine combination) but before the application of activation ³

• Steps

  • Covariate shift to standard position

  $$u_i=\frac{z_i-{\mu }_B}{\sqrt{{\sigma }_B^2+ϵ}}$$

  • Shift to right position

  $$\widehat{z_i}=\gamma u_i+\beta$$

Backpropagation

• The outputs are now functions of ${\mu }_B$ and ${\sigma }_B^2$ which are functions of the entire minibatch

$$\mathrm{Div}(MB)=\frac{1}{B}\sum _t\mathrm{Div}\left(Y_t\left(X_t,{\mu }_B,{\sigma }_B^2\right),d_t\left(X_t\right)\right)$$

• The divergence for each $Y_t$ depends on all the $X_t$ within the mini-batch
  • Is a vector function over the mini-batch
• Using influence diagram to caculate derivatives⁴

• Goal

  • We need to caculate the learnable parameters $\frac{dDiv}{\gamma },\frac{dDiv}{\beta }$, and the affine combination $\frac{dDiv}{z_i}$

  $$\frac{\partial Div}{\partial z_i}=\frac{\partial Div}{\partial u_i}\cdot \frac{\partial u_i}{\partial z_i}+\frac{\partial Div}{\partial {\sigma }_B^2}\cdot \frac{\partial {\sigma }_B^2}{\partial z_i}+\frac{\partial Div}{\partial {\mu }_B}\cdot \frac{\partial {\mu }_B}{\partial z_i}$$

  • So we need extra $\frac{\partial D i v}{\partial u_{i}},\frac{\partial D i v}{\partial \sigma_{B}^{2}},\frac{\partial D i v}{\partial \mu_{B}} $

• Preparation

$${\mu }_B=\frac{1}{B}\sum _{i=1}^B{z}_i{\sigma }_B^2=\frac{1}{B}\sum _{i=1}^B{\left(z_i-{\mu }_B\right)}^2$$

$$u_i=\frac{z_i-{\mu }_B}{\sqrt{{\sigma }_B^2+ϵ}}\widehat{z_i}=\gamma u_i+\beta$$

• For the first term $\frac{\partial Div}{\partial u_i}\cdot \frac{\partial u_i}{\partial z_i}$

  • First caculate $\frac{dDiv}{\gamma },\frac{dDiv}{\beta }$
    $$\frac{dDiv}{d\beta }=\frac{dDiv}{d\hat{z}}\frac{dDiv}{d\gamma }=u\frac{dDiv}{d\hat{z}}$$

  • $\frac{\partial u_i}{\partial z_i}=\frac{1}{\sqrt{{\sigma }_B^2+ϵ}}$, so the first term = $\frac{\partial Div}{\partial u_i}\cdot \frac{1}{\sqrt{{\sigma }_B^2+ϵ}}$

• For the second term $\frac{\partial Div}{\partial {\sigma }_B^2}\cdot \frac{\partial {\sigma }_B^2}{\partial z_i}$

  • Caculate $\frac{\partial Div}{\partial {\sigma }_B^2}$

  $$\frac{\partial Div}{\partial {\sigma }_B^2}=\sum \frac{\partial Div}{\partial u_i}\frac{\partial u_i}{\partial {\sigma }_B^2}$$

  $$\frac{\partial Div}{\partial {\sigma }_B^2}=\frac{-1}{2}{\left({\sigma }_B^2+ϵ\right)}^{-3/2}\sum _{i=1}^B\frac{\partial Div}{\partial u_i}\left(z_i-{\mu }_B\right)$$

  • And $\frac{\partial {\sigma }_B^2}{\partial z_i}$

  $$\frac{\partial {\sigma }_B^2}{\partial z_i}=\frac{2\left(z_i-{\mu }_B\right)}{B}$$

  • So the second term = $\frac{\partial Div}{\partial {\sigma }_B^2}\cdot \frac{2\left(z_i-{\mu }_B\right)}{B}$

• Finally for the third term $\frac{\partial Div}{\partial {\mu }_B}\cdot \frac{\partial {\mu }_B}{\partial z_i}$

  • Caculate $\frac{\partial Div}{\partial {\mu }_B}$

  $$\frac{\partial Div}{\partial {\mu }_B}=\sum \frac{\partial Div}{\partial {\mu }_i}\frac{\partial {\mu }_i}{\partial {\mu }_B}+\frac{\partial Div}{\partial {\sigma }_B^2}\frac{\partial {\sigma }_B^2}{\partial {\mu }_B}$$

  $$\frac{\partial Div}{\partial {\mu }_B}=\left(\sum _{i=1}^B\frac{\partial Div}{\partial u_i}\cdot \frac{-1}{\sqrt{{\sigma }_B^2+ϵ}}\right)+\frac{\partial Div}{\partial {\sigma }_B^2}\cdot \frac{{\sum }_{i=1}^B-2\left(z_i-{\mu }_B\right)}{B}$$

  • The last term is zero, and because ${\mu }_z=\frac{1}{B}\sum z_i$

  $$\frac{\partial {\mu }_B}{\partial z_i}=\frac{1}{B}$$

  • So the third term = $\frac{\partial Div}{\partial {\mu }_B}\cdot \frac{1}{B}$

• Overall

$$\frac{\partial Div}{\partial z_i}=\frac{\partial Div}{\partial u_i}\cdot \frac{1}{\sqrt{{\sigma }_B^2+ϵ}}+\frac{\partial Div}{\partial {\sigma }_B^2}\cdot \frac{2\left(z_i-{\mu }_B\right)}{B}+\frac{\partial Div}{\partial {\mu }_B}\cdot \frac{1}{B}$$

$$\frac{\partial Div}{\partial {\sigma }_B^2}=\frac{-1}{2}{\left({\sigma }_B^2+ϵ\right)}^{-3/2}\sum _{i=1}^B\frac{\partial Div}{\partial u_i}\left(z_i-{\mu }_B\right)$$

$$\frac{\partial Div}{\partial {\mu }_B}=\frac{-1}{\sqrt{{\sigma }_B^2+ϵ}}\sum _{i=1}^B\frac{\partial Div}{\partial u_i}$$

Inference

• On test data, BN requires ${\mu }_B$ and ${\sigma }_B^2$
• We will use the average over all training minibatches

$${\mu }_{BN}=\frac{1}{\text{Nbatches}}\sum _{bat}{\mu }_B(\text{batch})$$

$${\sigma }_{BN}^2=\frac{B}{(B-1)Nbatches}\sum _{batch}{\sigma }_B^2(\text{batch})$$

• Note: these are neuron-specific
  • ${\mu }_B(batch),{\sigma }_{B}batch$ are obtained from the final converged network
  • The 𝐵/(𝐵 − 1) term gives us an unbiased estimator for the variance

What can it do

• Improves both convergence rate and neural network performance
  • Anecdotal evidence that BN eliminates the need for dropout
• To get maximum benefit from BN, learning rates must be increased and learning rate decay can be faster
  • Since the data generally remain in the high-gradient regions of the activations
  • e.g. For sigmoid function, move data to the linear part, the gradient is high
• Also needs better randomization of training data order

Smoothness

• Smoothness through network structure
  • MLPs are universal approximators
  • For a given number of parameters, deeper networks impose more smoothness than shallow&wide ones
  • Each layer restricts the shape of the function
• Smoothness through weight constrain

Regularizer

• The "desired” output is generally smooth
  • Capture statistical or average trends
• Overfitting
  • But an unconstrained model will model individual instances instead
  • Why overfitting?⁵
• Using a sigmoid activation, as $|w|$ increases, the response becomes steeper

• Constraining the weights to be low will force slower perceptrons and smoother output response
• Regularized training: minimize the loss while also minimizing the weights

$$L\left(W_1,W_2,\dots ,W_K\right)=\mathrm{Loss}\left(W_1,W_2,\dots ,W_K\right)+\frac{1}{2}\lambda \sum _k{\Vert W_k\Vert }_2^2$$

• $\lambda$ is the regularization parameter whose value depends on how important it is for us to want to minimize the weights
• Increasing assigns greater importance to shrinking the weights
  • Make greater error on training data, to obtain a more acceptable network

Dropout

• “Dropout” is a stochastic data/model erasure method that sometimes forces the network to learn more robust models

• Bagging method

  • Using ensemble classifiers to improve prediction

• Dropout

  • For each input, at each iteration, “turn off” each neuron with a probability $1-\alpha$
  • Also turn off inputs similarly

• Backpropagation is effectively performed only over the remaining network

  • The effective network is different for different inputs
  • Effectively learns a network that averages over all possible networks (Bagging)

• Dropout as a mechanism to increase pattern density

  • Dropout forces the neurons to learn “rich” and redundant patterns
  • E.g. without dropout, a noncompressive layer may just “clone” its input to its output
  • Transferring the task of learning to the rest of the network upstream

Implementation

• The expected output of the neuron is
  $$y_i^{(k)}=\alpha \sigma \left(\sum _j{w}_{ji}^{(k)}{y}_j^{(k-1)}+b_i^{(k)}\right)$$

• During test, push the a to all outgoing weights

$$\begin{array}{rl}{z}_i^{(k)}& =\sum _j{w}_{ji}^{(k)}{y}_j^{(k-1)}+b_i^{(k)}\\ & \\ & =\sum _j{w}_{ji}^{(k)}\alpha \sigma \left(z_j^{(k-1)}\right)+b_i^{(k)}\\ & \\ & =\sum _j\left(\alpha w_{ji}^{(k)}\right)\sigma \left(z_j^{(k-1)}\right)+b_i^{(k)}\end{array}$$

• So $W_{test}=\alpha W_{trained}$
  • Instead of multiplying every output by all weights by $\alpha$, multiply all weight by $\alpha$
• Alternate implementation
  • During training, replace the activation of all neurons in the network by ${\alpha }^{-1}\sigma (.)$
  • Use $\sigma (.)$ as the activation during testing, and not modify the weights

More tricks

• Obtain training data
  • Use appropriate representation for inputs and outputs
  • Data Augmentation
• Choose network architecture
  • More neurons need more data
  • Deep is better, but harder to train
• Choose the appropriate divergence function
  • Choose regularization
• Choose heuristics
  • batch norm, dropout …
• Choose optimization algorithm
  • Adagrad / Adam / SGD
• Perform a grid search for hyper parameters (learning rate, regularization parameter, …) on held-out data
• Train
  • Evaluate periodically on validation data, for early stopping if required

---

1. A good summary of recent optimizers can be seen in here. ↩︎

2. Animations for optimization algorithms ↩︎

3. Batch normalization in Neural Networks ↩︎

4. A simple and clear demostration of 2 variables in a single network ↩︎

5. The perceptrons in the network are individually capable of sharp changes in output ↩︎