Machine Learning:Gradient Descent

Firstly,we need to know the basic Gradient Descent:
$${\theta }^{\ast }=arg\underset{\theta }{\min }L(\theta )$$
($L$ is loss function,and $\theta$ is parameter)
We suppose that $\theta$ has two variables{θ1,θ2}\{\theta_1,\theta_2\}{θ1​,θ2​}.Randomly start at ${\theta }^0=\left[\begin{array}{c}{\theta }_1^0\\ {\theta }_2^0\end{array}\right]$.And we can get
$$\left[\begin{array}{c}{\theta }_1^1\\ {\theta }_2^1\end{array}\right]=\left[\begin{array}{c}{\theta }_1^0\\ {\theta }_2^0\end{array}\right]-\eta \left[\begin{array}{c}\partial L({\theta }_1^0)/\partial {\theta }_1\\ \partial L({\theta }_2^0)/\partial {\theta }_2\end{array}\right]$$
We know that
$$\nabla L(\theta )=\left[\begin{array}{c}\partial L({\theta }_1)/\partial {\theta }_1\\ \partial L({\theta }_2)/\partial {\theta }_2\end{array}\right]$$
So we can get
$${\theta }^1={\theta }^0-\eta \nabla L({\theta }^0)$$

Tuning your learning rate

Learning rate is $\eta$.If $\eta$ is small,this procedure may consume too much time.But we can get the right $\theta$ in the end.If $\eta$ is large,this procedure may not get the result in the end.And if parameters more than 3,loss function can not be visualized.We can use ${\theta }^N-Loss$ to observe

Adaptive Learning Rates

There is a useful way to deal with
$${\eta }^t=\eta /\sqrt{t+1}$$

Adagrad

Root mean square.We can show examples to understand.
$$LR={\eta }^t/{\sigma }^t$$
If t=0:
$${\sigma }^0=\sqrt{(g^0{)}^2}$$
And
$$g^n=\nabla L({\theta }^n)$$
if t=1:
$${\sigma }^1=\sqrt{\frac{1}{2}[(g^0{)}^2+(g^1{)}^2]}$$
Therefore we can get:
$${\sigma }^t=\sqrt{\frac{1}{t+1}\sum _{i=0}^t(g^i{)}^2}$$
$$\frac{{\eta }^t}{{\sigma }^t}=\frac{\eta }{\sqrt{{\sum }_{i=0}^t(g^i{)}^2}}$$
Finally we can get Gradient Descent:
$$w^{t+1}=w^t-\frac{\eta }{\sqrt{{\sum }_{i=0}^t(g^i{)}^2}}{g}^t$$

Larger Gradient,Larger step?

For a parameter, the greater the differential, the greater the step is correct.But for multiple parameters, differential comparison across parameters is wrong.

Second Derivative

Actually best step is
$$\frac{|firstderivative|}{secondderivative}$$
So we can not judge steps by first derivative.

Stochastic Gradient Descent

Loss function:
$$L=\sum _n({\hat{y}}^n-(b+\sum w_i{x}_i^n){)}^2$$
Gradient Descent:
$$w^i=w^{i-1}-\eta \nabla L(w^{i-1})$$
In Stochastic Gradient Descent,we only choose one example $x^n$.Gradient Descent needs to understand all the examples, and then update.But Stochastic Gradient Descent traverses all the examples and performs updates.

Feature Scaling

Feature is input,like $x_1$ or $x_2$.Make different features have the same scaling.If $x_2$'s scaling is too large,we should shrink the range.

Why we need feature scaling?

If the value range of $x_1$ and $x_2$ is too large, it will cause the feature with a smaller value range to correspond to a larger parameter value range, that is to say, the parameter range is not equal. If the adaptive learning rate is not used, it will be difficult to perform a gradient descent.We need our Loss Function to be circle not ellipse.And alse if circle,gredient descent will directly point to center of circle.Ellipse will not.

How to feature scaling

Suppose there are many inputs, like $x^1$, $x^2$,…, $x^n$. Then each input has a feature,like$x_1^1$,$x_2^1$.We suppose that $x_3^n$'s scaling is too large.We calculate mean $m_3$ and standard deviation ${\sigma }_i$.
$$x_3^n=\frac{x_3^n-m_3}{{\sigma }_i}$$
The means of all dimensions are 0,and the variances are all 1.

Important!

Update the parameters may not reduce loss. And gradient descent may not only stuck at local minima,but also saddle point.Differential value equal to 0.In reality, we may choose to end the gredient descent when the Loss Function decreases slowly. Thinking that this will be close to the local minima, in fact, it may be higher than we think.