Goodfellow, Bengio, Couville (2016) "Deep Learning", Chap. 7

Regularization for Deep Learning
Goodfellow, Bengio, & Courville (2016) Deep Learning, Chap 7.
Shigeru ONO (Insight Factory)
DL 読書会: 2020/08
Shigeru ONO (Insight Factory) DL Chap.7 DL 読書会: 2020/08 1 / 40

TOC
1 7.1 Parameter Norm Penalties
2 7.2 Norm Penalties as Constrained Optimization
3 7.3 Regularization and Under-Constrained Problems
4 7.4 Dataset Augmentation
5 7.5 Noise Robustness
6 7.6 Semi-Supervised Learning
7 7.7 Multitask Learning
8 7.8 Early Stopping
9 7.9 Parameter Tying and Parameter Sharing
10 7.10 Sparse Representation
11 7.11 Bagging and Other Ensemble Methods
12 7.12 Dropout
13 7.13 Adversarial Training
14 7.14 Tangent Distance, Tangent Prop and Manifold Tangent Classifier

(introduction)
Regularization:
any modification we make to a learning algorithm that is intended to reduce
its generalization error
possibly at the expense of increasing training error
In the context of DL, most regularization strategies are based on regularizing
estimators
Possible situations (See Chap.5) :
(1) the model family excluded the true DGP (underfitting)
(2) the model family matched the true DGP
(3) the model family included the true DGP but also many other possible DGP
(overfitting)
The goal of regularization is to take the model from (3) into (2). But...
In most applications of DL, the true DGP is outside the model family (=(1)).
Controlling the complexity of the model is not to find the model of the right
size, but to find the model with appropriate regularization in which
generalization error is minimized.

7.1 Parameter Norm Penalties
Adding a parameter norm penalty Ω(θ) to the objective function J.
˜J(θ; X, y) = J(θ; X, y) + αΩ(θ)
α(≥ 0): weight of the relative contribution of Ω.
For NN, we typically choose Ω that penalizes only w (the weights of the
aﬀine transformation at each layer).
It is reasonable to use the same α at all layers.

7.1.1 L2
Parameter Regularization
Ω(θ) =
1
2
||w||2
aka. weight decay, ridge regression, Tikhonov regularization.
Bayesian interpretation: MAP inference with a Gaussian prior on the weights.
(See 5.6.1)
Total objective function:
˜J(w; X, y) = J(w; X, y) +
α
2
w⊤
w
Parameter gradient:
∇w
˜J(w; X, y) = αw + ∇wJ(w; X, y)
What happens in a single gradient step? ... The learning rule is modified to
shrink w by a constant factor.
w ← (1 − ϵα)w − ϵ∇wJ(w; X, y)

7.1.1 L2
What happens over the entire course of training? (in general)
Unregularized:
Let w∗
be the weights which minimize the unregularized objective function:
w∗
= arg min
w
J(w).
Make a quadratic approximation to the J(w) in the neighborhood of w∗
:
ˆJ(θ) = J(w∗
) +
1
2
(w − w∗
)⊤
H(w − w∗
)
where H is the Hessian matrix of J with respect to w evaluated at w∗
.
The minimum of ˆJ occurs where its gradient ∇w
ˆJ(w) = H(w − w∗
) is 0.

7.1.1 L2
(Cont’d)
Regularized:
Let ˜w be the weights with minimize the regularized objective function ˜J.
The minimum of ˜J occurs where α˜w + H(w − w∗
) = 0.
It follows that ˜w = (H + αI)−1
Hw∗
H is real and symmetric. We can have a eigenvalue decomposition
H = QΛQ⊤
.
˜w = Q(Λ + αI)−1
ΛQ⊤
w∗
. i.e. The weight decay rescales w∗
along the axes
defined by the eigenvector of H.

7.1.1 L2
What happens over the entire course of training? (in the case of linear regression)
Unregularized:
Cost function: (Xw − y)⊤
(Xw − y)
Solution: w = (X⊤
X)−1
X⊤
y
Regularized:
Cost function: (Xw − y)⊤
(Xw − y) + 1
2 αw⊤
w
Solution: w = (X⊤
X + αI)−1
X⊤
y.
i.e. The regularization cause the algorithm to ”perceive” that X has higher
variance (than the variance it really has).

7.1.1 L2

7.1.2 L1
Regularization
Ω(θ) = ||w||1 =
∑
i
|wi|
Total objective function :
˜J(w; X, y) = J(w; X, y) + α||w||1
Parameter gradient:
∇w
˜J(w; X, y) = αsign(w) + ∇wJ(w; X, y)
It does not admit clean algebraic solution.
For simple linear model with a quadratic cost function,
∇w
ˆJ(w) = H(w − w∗
)

7.1.2 L1
Regularization
(Cont’d)
Assume the Hessian is diagonal, H = diag([H11, . . . , Hnn]) (i.e. no correlation
between the input features)
Then we have a quadratic approximation of the cost function:
ˆJ(w; X, y) = J(w∗
; X, y) +
∑
i
(
1
2
Hii(wi − w∗
i )2
+ α|wi|
)
The solution is:
wi = sign(w∗
i ) max
(
|w∗
i | −
α
Hii
, 0
)
Consider the situation where w∗
i > 0 for all i. Then
When w∗
i ≤ α
Hii
, the optimal value is wi = 0.
When w∗
i > α
Hii
, the optimal value is just shifted by a distance α
Hii
.

7.1.2 L1
Regularization
(Cont’d)
In short, the solution is more sparse (i.e. some parameter have an optimal
value of zero).
It has been used as a feature selection mechanism. E.g. LASSO
Bayesian interpretation: MAP inference with a isotropic Laplace prior on the
weights.

7.2 Norm Penalties as Constrained Optimization
We can think of the penalties as constraints.
Cost function:
˜J(θ; X, y) = J(θ; X, y) + αΩ(θ)
If we wanted to constrain as Ω(θ) < k, we could construct a generalized
Lagrangian
L(θ, α; X, y) = J(θ; X, y) + α(Ω(θ) − k)
The solution is
θ∗
= arg min
θ
max
α,α≥0
L(θ, α)
we can fix α as its optimal value α∗
:
θ∗
= arg min
θ
L(θ, α∗
) = arg min
θ
J(θ; X, y) + α∗
Ω(θ)
This is same as the problem of minimizing ˜J.

7.2 Norm Penalties as Constrained Optimization
Sometimes we may with to use explicit constraints rather than penalties:
when we know the appropriate value of k
when the penalties can cause optimization to get stuck in local minima
corresponding to small θ.
when we with to impose some stability on the optimization procedure
Approach:
Srebro & Shraibman (2005): constraining the norm of each column of the
weight matrix of a layer

7.3 Regularization and Under-Constrained Problems
Sometimes regularization is necessary for ML problems to be properly defined.
when the problem depends on (X⊤
X)−1
but X⊤
X is singular.
when the problem has no closed form solution. E.g., logistic regression
applied to a problem where the class are linear separable. If weight w can
achieve perfect classification, 2w will also achieve with higher likelihood.

7.4 Dataset Augmentation
Idea: Create fake data and add it to the training set.
an effective technique particularly for object recognition. E.g. translating the
training images a few pixels in each direction.
Injecting noise in the input to a NN. It can improve the robustness of NNs.

7.5 Noise Robustness
Idea: Add noise to the weights.
It can be interpreted as a stochastic implementation of Bayesian inference
over the weight.
Noise reflect our uncertainty on the model weights.
It can also be interpreted as equivalent to a more traditional form of
regularization.
Consider we wish to train a function ˆy(x) using the least-square cost function
J = Ep(x,y)[(ˆy(x) − y)2
]
Assume that we also include a random perturbation ϵw ∼ N(ϵ; 0, ηI) of the
network weights.
The objective function becomes ˜JW = Ep(x,y,ϵW)[(ˆyeW (x) − y)2
]
For small η, it is equivalent to J with a regularization term
ηEp(x,y)[||∇Wˆy(x)||2
].
It push the model into regions where the model is relatively insensitive to small
variations in the weights, finding points that are not merely minima, but
minima surrounded by flat regions.

7.5.1 Injecting Noise at the Output Targets
Idea: Explicitly model the noise on the y labels.
label smoothing: regularize a model based on a softmax with k output values
by replacing classification target
0 with ϵ/(k − 1)
1 with 1 − ϵ

7.6 Semi-Supervised Learning
Idea: Use both unlabeled example (from P(x)) and labeled example (from P(x, y))
in order to estimate P(y|x)
In the context of DL, semi-supervised learning usually refers to learning a
representation h = f(x).
The goal is to learn a representation so that examples from the same class
have similar representations.
Construct models in which a generative model of either P(x) or P(x, y) shares
parameters with a discriminative model of P(y|x)
One can find a better trade-off of two types of criterion:
The supervised criterion: − log P(y|x)
The unsupervised (generative) criterion: − log P(x) or − log P(x, y)

7.7 Multitask Learning
Idea: Pool the examples arising out of several tasks
The model can be divided into two parts:
Task-specific parameters
Generic parameters, shared across the tasks
It can improve generalization and generalization error bounds

7.7 Multitask Learning

7.8 Early Stopping
Idea : Obtain a model with the parameters at the point in time with the lowest
validation set error (rather than with the latest parameters in the training process)
the most commonly use form of regularization in DL
can be interpreted as a hyperparameter (the number of training steps)
selection algorithm
requires a validation set, which is not fed to the model
One can perform extra training (where all training data is used) after initial
learning (with early stopping). Two basic strategies:
Initialize the model again and retrain on all the data (for the same number of
steps as the first round)
Keep the parameter and continue training (but now using all the data). It is
not as well behaved.

7.8 Early Stopping
How early stopping acts as regularizer:
Restricting both the number of iterations and the learning rate limit the
volume of parameter space reachable from the initial parameter value.
In a simple linear model with a quadratic error function and simple gradient
decent, early stopping is equivalent to L2
regularization. [...skipped...]

7.8 Early Stopping

7.9 Parameter Tying and Parameter Sharing
Sometimes we may know there should be some dependencies between the
parameters.
Parameter Tying:
E.g. two models performing the same classification task but with different
input distributions:
ˆy(A)
= f(w(A))
, x), ˆy(B)
= f(w(B))
, x)
We believe the model parameters should be close to each other
We can use a penalty Ω(w(A)
, w(B)
) = ||w(A)
− w(B)
||2
Parameter Sharing:
force sets of parameters to be equal
can lead to significant reduction of memory
The most popular use: convolutional neural network (CNNs) (See Chap.9)

7.10 Sparse Representation
Idea: place a penalty on the activation of the unit (rather than on the parameters)
Norm penalty regularization of representation:
h: sparse representation of the data x
add a norm penalty on the representation Ω(h) to the loss function J:
˜J(θ; X, y) = J(θ; X, y) + αΩ(h)
We can use L1
penalty Ω(h) = ||h||1 or other types of penalties
Orthogonal matching pursuit (OMP-k):
encodes x with h that solves the constrained optimization problem
arg min
h,||h||0<k
||x − Wh||2
where ||h||0 is the number of nonzero entries of h
OMP-1 can be a very effective feature extractor for DL

7.11 Bagging and Other Ensemble Methods
Ensemble methods
combine several models (trained separately) in order to reduce generalization
error
an example of a general strategy called model averaging
On average, the ensemble will perform at least as well as any of its members.
If the members make independent errors, the ensemble will perform
significantly better.
Bagging (bootstrap aggregating)
construct k different datasets of same size by sampling with replacement
from the original dataset
Model i is trained on data set i
Boosting:
construct an ensemble with higher capacity then individual models
Boosting of NN: incrementally add NN to the ensemble

7.12 Dropout
Background:
Bagging involves training multiple models and evaluating them on each test
example.
This seems impractical when each model is a large NN.
Dropout can be thought of as a method of making bagging practical.
What is Dropout?
make all subnetworks that can be formed by removing nonoutput units from
an base network
In many cases, we can remove a unit by multiplying its output value by zero.
Let µ be a vector of binary mask, which is applied to all the input and hidden
units.
train them with a minibatch-based algorithm
Each time we load an example into a minibatch, we randomly sample µ and
apply it.
Typically, an input unit is included with probability 0.8, and a hidden unit is
included with 0.5.
Run forward propagation, back-propagation, and the learning update.

7.12 Dropout

7.12 Dropout
How to make a prediction:
At training time, µ is sampled from the probability distribution p(µ)
Each submodel defined by µ defines a probability distribution p(y|x, µ)
To make a prediction from all submodels, we can use arithmetic mean:∑
µ p(µ)p(y|x, µ)
But geometric mean performs better. Let ˜pensemble(y|x) be the geometric
mean of p(y|x, µ).
˜p(y|x) is not guaranteed to be a probability distribution. We must
renormalize:
pensemble(y|x) =
˜pensemble(y|x)
∑
y′ ˜pensemble(y′|x)

7.12 Dropout
Weight scaling inference rule:
We can approximate pensemble by evaluating p(y|x) in one model
This model uses all units, but with the weights going out of unit i multiplied
by the probability of including unit i
if an inclusion probability of a unit is 1/2, the weight of the unit is multiplied
by 1/2 at the end of training, or the states of the unit is multiplied by 2 during
training
There is not yet any theoretical argument for this rule, but empirically it
performs well

7.12 Dropout
Advantages of dropout:
very computationally cheap
it does not significantly limit the type of model or training procedure
Limitations:
it reduces the effective capacity of a model. To offset this effect, we must
increase the size of the model.
it is less effective when extremely few labeled training examples are available.
When additional unlabeled data is available, unsupervised feature learning
can gain an advantage over dropout.

7.12 Dropout
fast dropout:
analytical approximations to the sum over all submodels
more principled approach than the weight scaling inference rule.
Interpretation of dropout:
an experiments using ”dropout boosting”
use exactly the same mask noise as dropout
trains the entire ensemble to jointly (not independently) maximize the
log-likelihood on the training set
shows almost no regularization effect
This demonstrates that dropout is a type of bagging. Dropout in itself have
no robustness to noise.

7.12 Dropout
Other approaches inspired by dropout:
DropConnect: each product between a single scalar weight and a single
hidden unit state is considered a unit that can be dropped
Stochastic pooling: build ensembles of CNNs
real valued mask: multiplying the weights by µ ∼ N(1, I) can outperform
dropout
Another view of dropout:
Dropout regularize each hidden unit to be not merely a good feature but a
feature that is good in many context.
Masking can be seen as a form of highly intelligent, adaptive destruction of
the information content of the input (rather than destruction of the raw
input). It allows the model to make use of all the knowledge about the input
distribution which has acquired so far.

7.13 Adversarial Training
Adversarial example:
an input x′
near a data point x such that the model output is very different
at x′
In many cases, human observer cannot tell the difference between x and x′
One of the causes of these examples is excessive linearity in NN. The value of
a linear function can change very rapidly if it has numerous inputs.

Adversarial Training:
training on adversarially perturbed examples from the training set
a way of explicitly introducing a local constancy prior into NN
Virtual adversarial example:
Suppose the model assigns some label ˆy at a point x which has no true label.
We can seek an adversarial example x′
that causes the model to output a
label y′
(̸= ˆy)
We can train the model to assign the same label to x and x′
This encourages the model to learn a function which is robust to small change
This provide a means of semi-supervised learning

7.14 Tangent Distance, Tangent Prop and Manifold
Tangent Classifier
Manifold hypothesis:
the data lies near a low-dimensional manifold
Tangent distance algorithm
non-parametric nearest neighbor algorithm, where the distance between
points x1 and x2 is the distance between the manifolds M1 and M2 to which
they respectively belong
approximate Mi by its tangent plane at xi
The user has to specify the tangent vectors

7.14 Tangent Distance, Tangent Prop and Manifold
Tangent Classifier
Tangent prop algorithm:
[...skipped...]
double backprop:
[...skipped...]

Goodfellow, Bengio, Couville (2016) "Deep Learning", Chap. 7

More Related Content

What's hot (20)

Similar to Goodfellow, Bengio, Couville (2016) "Deep Learning", Chap. 7 (20)

More from Ono Shigeru (6)

Recently uploaded (20)

Goodfellow, Bengio, Couville (2016) "Deep Learning", Chap. 7