CMU 11-785 L04 Backpropagation_cmu 11-785作业-CSDN博客

本文链接：https://blog.csdn.net/crazy_scott/article/details/105666249

本文深入探讨了深度学习中网络训练的过程，包括输入输出对的设定、使用一热向量表示输出、以及如何通过不同类型的损失函数如L2范数、交叉熵来衡量模型预测与实际标签之间的差距。特别讲解了反向传播算法，从输出层到输入层逐层计算梯度，并详细解析了在乘法组合层和向量激活函数下梯度的计算方式。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Problem setup

Input-output pairs: not to mention
Representing the output: one-hot vector
- $y_{i}=\frac{\exp \left(z_{i}\right)}{\sum_{j} \exp \left(z_{j}\right)}$
- two classes of softmax = sigmoid
Divergence: must be differentiable
- For real-valued output vectors, the (scaled) $L_2$ divergence
  - $\operatorname{Div}(Y, d)=\frac{1}{2}\|Y-d\|^{2}=\frac{1}{2} \sum_{i}\left(y_{i}-d_{i}\right)^{2}$
- For binary classifier
  - $\operatorname{Div}(Y, d)=-\operatorname{dlog} Y-(1-d) \log (1-Y)$
  - Note: the derivative is not zero even $d = Y$ , but it can converge very quickly
- For multi-class classification
  - $\operatorname{Div}(Y, d)=-\sum_{i} d_{i} \log y_{i}=-\log y_{c}$
  - If $y_c < 1$ , the slope is negative w.r.t. $y_c$ , indicates increasing $y_c$ will reduce divergence

Train the network

Distributed Chain rule

$y=f\left(g_{1}(x), g_{1}(x), \ldots, g_{M}(x)\right)$

$\frac{d y}{d x}=\frac{\partial f}{\partial g_{1}(x)} \frac{d g_{1}(x)}{d x}+\frac{\partial f}{\partial g_{2}(x)} \frac{d g_{2}(x)}{d x}+\cdots+\frac{\partial f}{\partial g_{M}(x)} \frac{d g_{M}(x)}{d x}$

Backpropagation

在这里插入图片描述

For each layer: we caculate $∂Div∂yi\frac{\partial D i v}{\partial y_{i}}$ , $∂Dicv∂z\frac{\partial Dicv}{\partial z}$ , and $∂Div∂wij\frac{\partial Div}{\partial w_{ij}}$
For ouput layer
- It is easy to caculate $∂Div∂yi(N)\frac{\partial D i v}{\partial y_{i}^{(N)}}$
- So: $∂Div∂zi(N)=fN′(zi(N))∂Div∂yi(N)\frac{\partial D i v}{\partial z_{i}^{(N)}}=f_{N}^{\prime}\left(z_{i}^{(N)}\right) \frac{\partial D i v}{\partial y_{i}^{(N)}}$
- $∂Div∂wij(N)=∂zj(N)∂wij(N)∂Div∂zj(N)\frac{\partial D i v}{\partial w_{ij}^{(N)}}=\frac{\partial z_{j}^{(N)}}{\partial w_{ij}^{(N)}} \frac{\partial D i v}{\partial z_{j}^{(N)}}$ , where $∂zj(N)∂wij(N)=yi(N)\frac{\partial z_{j}^{(N)}}{\partial w_{ij}^{(N)}} = y_i^{(N)}$
Pass on
- $zj(N)=∑iwij(2)yi(v−1)z_{j}^{(N)}=\sum_{i} w_{i j}^{(2)} y_{i}^{(v-1)}$ , so $∂zj(N)∂y1(N−1)=wij(N)\frac{\partial z_{j}^{(N)}}{\partial y_{1}^{(N-1)}} = w_{ij}^{(N)}$
- $∂Div∂yi(N−1)=∑jwij(N)∂Div∂zj(N)\frac{\partial D i v}{\partial y_{i}^{(N-1)}}=\sum_{j} w_{i j}^{(N)} \frac{\partial D i v}{\partial z_{j}^{(N)}}$
- $∂Div∂zi(N−1)=fN−1′(zi(N−1))∂Div∂yi(N−1)\frac{\partial D i v}{\partial z_{i}^{(N-1)}}=f_{N-1}^{\prime}(z_{i}^{(N-1)}) \frac{\partial D i v}{\partial y_{i}^{(N-1)}}$
- $∂Div∂wij(N−1)=yi(N−2)∂Div∂zj(N−1)\frac{\partial D i v}{\partial w_{i j}^{(N-1)}}=y_{i}^{(N-2)} \frac{\partial D i v}{\partial z_{j}^{(N-1)}}$

在这里插入图片描述

Special case

Vector activations

Vector activations: all outputs are functions of all inputs

在这里插入图片描述

So the derivatives need to change a little
$\frac{\partial D i v}{\partial z_{i}^{(k)}}=\sum_{j} \frac{\partial D i v}{\partial y_{j}^{(k)}} \frac{\partial y_{j}^{(k)}}{\partial z_{i}^{(k)}}$
Note: derivatives of scalar activations are just a special case of vector activations:
$\frac{\partial y_{j}^{(k)}}{\partial z_{i}^{(k)}}=0 \text { for } i \neq j$
For example, Softmax:

$y_{i}^{(k)}=\frac{\exp \left(z_{i}^{(k)}\right)}{\sum_{j} \exp \left(z_{j}^{(k)}\right)}$

$\frac{\partial D i v}{\partial z_{i}^{(k)}}=\sum_{j} \frac{\partial D i v}{\partial y_{j}^{(k)}} \frac{\partial y_{j}^{(k)}}{\partial z_{i}^{(k)}}$

$KaTeX parse error: Got function '\newline' with no arguments as argument to '\left' at position 1: \̲n̲e̲w̲l̲i̲n̲e̲$

Using Keonecker delta $δij=1\delta_{i j}=1$ if $\quad 0$ if $\neq j$

$\frac{\partial D i v}{\partial z_{i}^{(k)}}=\sum_{j} \frac{\partial D i v}{\partial y_{j}^{(k)}} y_{i}^{(k)}\left(\delta_{i j}-y_{j}^{(k)}\right)$

Multiplicative networks

Some types of networks have multiplicative combination(instead of additive combination)
Seen in networks such as LSTMs, GRUs, attention models, etc.

在这里插入图片描述

So the derivatives need to change

$\frac{\partial D i v}{\partial o_{i}^{(k)}}=\sum_{j} w_{i j}^{(k+1)} \frac{\partial D i v}{\partial z_{j}^{(k+1)}}$

$\frac{\partial D i v}{\partial y_{j}^{(k-1)}}=\frac{\partial o_{i}^{(k)}}{\partial y_{j}^{(k-1)}} \frac{\partial D i v}{\partial o_{i}^{(k)}}=y_{l}^{(k-1)} \frac{\partial D i v}{\partial o_{i}^{(k)}}$

A layer of multiplicative combination is a special case of vector activation

Non-differentiable activations

Activation functions are sometimes not actually differentiable
- The RELU (Rectified Linear Unit)
  - And its variants: leaky RELU, randomized leaky RELU
- The “max” function
Subgradient
- $\left(f(x)-f\left(x_{0}\right)\right) \geq v^{T}\left(x-x_{0}\right)$
- The subgradient is a direction in which the function is guaranteed to increase
- If the function is differentiable at $x$ , the subgradient is the gradient
- But gradient is not always the subgradient though

Vector formulation

Define the vectors:

在这里插入图片描述

Forward pass

在这里插入图片描述

Backward pass

Chain rule
- $y=f(g(x))\mathbf{y}=\boldsymbol{f}(\boldsymbol{g}(\mathbf{x}))$
- Let $z = g (x)$ , $y = f (z)$
- So $Jy(x)=Jy(z)Jz(x)J_{\mathbf{y}}(\mathbf{x})=J_{\mathbf{y}}(\mathbf{z}) J_{\mathbf{z}}(\mathbf{x})$
For scalar functions:
- $D = f (W y + b)$
- Let $z = W y + b$ , $D = f (z)$
- $∇xD=∇z(D)Jz(x)\nabla_{x} D = \nabla_z(D)J_z(x)$
So for backward process
- $∇ZNDiv=∇YDiv∇ZNY\nabla_{Z_N} Div = \nabla_Y Div \nabla_{Z_N}Y$
- $\nabla_{y_{N-1}}Div = \nabla_{Z_N} Div \nabla_{y_{N-1}} z_N $
- $∇WNDiv=yN−1∇ZNDiv\nabla_{W_N} Div = y_{N-1} \nabla_{Z_N} Div$
- $∇bNDiv=∇ZNDiv\nabla_{b_N} Div = \nabla_{Z_N} Div$
For each layer
- First compute $∇yDiv\nabla_{y} Div$
- Then compute $∇zDiv\nabla_{z}Div$
- Finally $∇WDiv\nabla_{W} Div$ , $∇WDiv\nabla_{W} Div$