深度学习学习笔记——C2W1-11——深度网络的权重初始化

Weight Initialization for Deep Networks深度网络的权重初始化

In the last video you saw how very deep neural networks can have the problems of vanishing and exploding gradients. It turns out that a partial solution to this, doesn't solve it entirely but helps a lot, is better or more careful choice of the random initialization for your neural network. To understand this, let's start with the example of initializing the ways for a single neuron, and then we're go on to generalize this to a deep network.

Let's go through this with an example with just a single neuron, and then we'll talk about the deep net later. So with a single neuron, you might input four features, x1 through x4, and then you have some a=g(z) and then it outputs some y. And later on for a deeper net, you know these inputs will be right, some layer a(l), but for now let's just call this x for now. So z is going to be equal to w1x1 + w2x2 +... + I guess WnXn. And let's set b=0 so, you know, let's just ignore b for now. So in order to make z not blow up and not become too small, you notice that the larger n is, the smaller you want Wi to be, right? Because z is the sum of the WiXi. And so if you're adding up a lot of these terms, you want each of these terms to be smal