Noisyactivationfunction资源-CSDN下载

需积分: 9 119 浏览量 2017-12-03 09:49:34 上传评论收藏 736KB PDF 举报

标题："Noisy activation function"，描述："Noisy activation function pdf 很好的论文。。。。。。。。"，标签："Noisy activation function"，部分内容：“NoisyActivationFunctionsCaglarGulcehre†(cid:93)MarcinMoczulski†(cid:5)MishaDenil†(cid:5)YoshuaBengio†(cid:93)†(cid:93)UniversityofMontreal†(cid:5)UniversityofOxford6...”，这些信息中蕴含的IT知识点如下：标题和描述中的"Noisy activation function"，即“噪声激活函数”，是一种在深度学习网络中使用的特殊激活函数。这种激活函数在激活函数中引入了噪声，使得梯度能够流动，即使在无噪声的情况下，该激活函数的应用可能导致梯度为零。这在神经网络的训练中，尤其在激活函数饱和导致梯度消失的情况下，提供了新的解决方案。描述中提到的“Noisy activation function pdf 很好的论文”，说明这份论文详细描述了噪声激活函数的原理和应用，是对该领域研究的重要贡献。在部分内容中，“Commonnonlinearactivationfunctionsusedin neuralnetworkscancausetrainingdifficulties due to the saturation behavior of the activation function, which may hide dependencies that are not visible to vanilla-SGD(using first order gradients only)”。这里提到了一种常见的非线性激活函数在神经网络中可能导致训练困难的问题，那就是激活函数的饱和行为可能会隐藏一些在使用仅一阶梯度的普通随机梯度下降（vanilla-SGD）中不可见的依赖性。 “Gatingmechanismsthatuse softly saturating activation functions to emulate the discrete switching of digital logic circuits are good examples of this”。这句话描述了使用软饱和激活函数来模拟数字逻辑电路的离散开关的门控机制，这是一个饱和激活函数可能导致训练困难的例子。 “Weproposetoexploit the injection of appropriate noise so that gradients may flow easily, even if the noiseless application of the activation function would yield zero gradient”。这部分提出了通过引入适当的噪声，使得即使在没有噪声的情况下，激活函数的应用将产生零梯度，梯度也可以流动，从而解决了训练困难的问题。 “Largenoisewilldominatethe noise-free gradient and allowstochastic gradient descent to explore more”。大噪声将主导无噪声梯度，允许随机梯度下降探索更多可能性。 “By adding noise only to the problematic parts of the activation function, we allow the optimization procedure to explore the boundary between the degenerate (saturating) and the well-behaved parts of the activation function”。这句话说明，仅在激活函数的有问题部分添加噪声，允许优化过程探索激活函数的退化（饱和）和良好行为部分之间的边界。此外，“Wealsoestablishconnectionstosimulated annealing, when the amount of noise is annealed down, making it easier to optimize hard objective functions”。这部分提到了将噪声激活函数与模拟退火算法建立联系，当噪声被降低时，优化困难的目标函数变得更易于优化。 “We find experimentally that replacing such saturating activation functions by noisy variants helps training in many contexts, yielding state-of-the-art or competitive results on different datasets and tasks, especially when training seems to be the most difficult, e.g., when curriculum learning is necessary to obtain good results”。这部分描述了实验发现，用噪声变体替换饱和激活函数在许多情况下有助于训练，对不同的数据集和任务产生最先进的或有竞争力的结果，特别是在训练似乎最困难的时候，例如当需要使用课程学习才能获得良好结果的情况下。总结来说，这篇文章提出的噪声激活函数，通过在激活函数中引入适当的噪声，解决了传统激活函数在训练过程中可能导致梯度消失的问题，提高了深度学习模型的训练效率和效果，特别是在处理复杂任务和数据集时表现尤为突出。

资源推荐

资源详情

资源评论

Noisy Activation Functions

Caglar Gulcehre

†]

GULCEHRC@IRO.UMONTREAL.CA

Marcin Moczulski

†

MARCIN.MOCZULSKI@STCATZ.OX.AC.UK

Misha Denil

†

MISHA.DENIL@GMAIL.COM

Yoshua Bengio

†]

BENGIOY@IRO.UMONTREAL.CA

†]

University of Montreal

†

University of Oxford

Abstract

Common nonlinear activation functions used in

neural networks can cause training difﬁculties

due to the saturation behavior of the activation

function, which may hide dependencies that

are not visible to vanilla-SGD (using ﬁrst order

gradients only). Gating mechanisms that use

softly saturating activation functions to emulate

the discrete switching of digital logic circuits are

good examples of this. We propose to exploit

the injection of appropriate noise so that the

gradients may ﬂow easily, even if the noiseless

application of the activation function would yield

zero gradient. Large noise will dominate the

noise-free gradient and allow stochastic gradient

descent to explore more. By adding noise only to

the problematic parts of the activation function,

we allow the optimization procedure to explore the

boundary between the degenerate (saturating) and

the well-behaved parts of the activation function.

We also establish connections to simulated

annealing, when the amount of noise is annealed

down, making it easier to optimize hard objective

functions. We ﬁnd experimentally that replacing

such saturating activation functions by noisy

variants helps training in many contexts, yielding

state-of-the-art or competitive results on different

datasets and task, especially when training seems

to be the most difﬁcult, e.g., when curriculum

learning is necessary to obtain good results.

1. Introduction

The introduction of the piecewise-linear activation functions

such as ReLU and Maxout (Goodfellow et al., 2013) units

had a profound effect on deep learning, and was a major

catalyst in allowing the training of much deeper networks. It

is thanks to ReLU that for the ﬁrst time it was shown (Glorot

et al., 2011) that deep purely supervised networks can be

trained, whereas using tanh nonlinearity only allowed to

train shallow networks. A plausible hypothesis about the

recent surge of interest on these piecewise-linear activation

functions (Glorot et al., 2011), is due to the fact that they

are easier to optimize with SGD and backpropagation than

smooth activation functions, such as

sigmoid

and

tanh

. The

recent successes of piecewise linear functions is particularly

evident in computer vision, where the ReLU has become the

default choice in convolutional networks.

We propose a new technique to train neural networks with

activation functions which strongly saturate when their input

is large. This is mainly achieved by injecting noise to the

activation function in its saturated regime and learning the

level of noise. Using this approach, we have found that it

was possible to train neural networks with much wider family

of activation functions than previously. Adding noise to the

activation function has been considered for ReLU units and

was explored in (Bengio et al., 2013; Nair & Hinton, 2010) for

feed-forward networks and Boltzmann machines to encourage

units to explore more and make the optimization easier.

More recently there has been a resurgence of interest in more

elaborated “gated” architectures such as LSTMs (Hochreiter

& Schmidhuber, 1997) and GRUs (Cho et al., 2014), but also

encompassing neural attention mechanisms that have been

used in the NTM (Graves et al., 2014), Memory Networks

(Weston et al., 2014), automatic image captioning (Xu et al.,

2015), video caption generation (Yao et al., 2015) and wide

areas of applications (LeCun et al., 2015). A common thread

running through these works is the use of soft-saturating

non-linearities, such as the sigmoid or softmax, to emulate

the hard decisions of digital logic circuits. In spite of its

success, there are two key problems with this approach.

Since the non-linearities still saturate there are problems

with vanishing gradient information ﬂowing through the

gates; and

2. since the non-linearities only softly saturate they do not

allow one to take hard decisions.

Although gates often operate in the soft-saturated regime

(Karpathy et al., 2015; Bahdanau et al., 2014; Hermann et al.,

2015) the architecture prevents them from being fully open or

closed. We follow a novel approach to address both of these

problems. Our method addresses the second problem through

the use of hard-saturating nonlinearities, which allow gates

arXiv:1603.00391v3 [cs.LG] 3 Apr 2016

Noisy Activation Functions

to make perfectly on or off decisions when they saturate.

Since the gates are able to be completely open or closed, no

information is lost through the leakiness of the soft-gating

architecture.

By introducing hard-saturating nonlinearities, we have

exacerbated the problem of gradient ﬂow, since gradients

in the saturated regime are now precisely zero instead of

being negligible. However, by introducing noise into the

activation function which can grow based on the magnitude

of saturation, we encourage random exploration.

At test time the noise in the activation functions can be

removed or replaced with the expectation, and as our

experiments show, the resulting deterministic networks

outperform their soft-saturating counterparts on a wide variety

of tasks, and allow to reach state-of-the-art performance by

simple drop-in replacement of the nonlinearities in existing

training code.

The technique that we propose, addresses the difﬁculty of

optimization and having hard-activations at the test time for

gating units and we propose a way of performing simulated

annealing for neural networks.

Hannun et al. (2014); Le et al. (2015) used ReLU activation

functions with simple RNNs. In this paper, we successfully

show that, it is possible to use piecewise-linear activation

functions with gated recurrent networks such as LSTM and

GRU’s.

2. Saturating Activation Functions

Deﬁnition 2.1. (Activation Function)

. An activation func-

tion is a function h : R → R that is differentiable almost

everywhere.

Deﬁnition 2.2. (Saturation). An activation function h(x)

with derivative h

(x) is said to right (resp. left) saturate if

its limit as x → ∞ (resp. x → −∞) is zero. An activation

function is said to saturate (without qualiﬁcation) if it both

left and right saturates.

Most common activation functions used in recurrent networks

(for example,

tanh

and

sigmoid

) are saturating. In particular

they are soft saturating, meaning that they achieve saturation

only in the limit.

Deﬁnition 2.3. (Hard and Soft Saturation). Let c be a

constant such that x > c implies h

(x) = 0 and left hard

saturates when x < c implies h

(x) = 0, ∀x. We say that

h(·)

hard saturates (without qualiﬁcation) if it both left and

right hard saturates. An activation function that saturates but

achieves zero gradient only in the limit is said to soft saturate.

We can construct hard saturating versions of soft saturating

activation functions by taking a ﬁrst-order Taylor expansion

about zero and clipping the results to an appropriate range.

For example, expanding tanh and sigmoid around 0, with

x ≈ 0

, we obtain linearized functions

and

tanh

and

sigmoid respectively:

sigmoid(x) ≈ u

(x) = 0.25x + 0.5 (1)

tanh(x) ≈ u

(x) = x. (2)

Clipping the linear approximations result to,

hard-sigmoid(x) = max(min(u

(x), 1), 0) (3)

hard-tanh(x) = max(min(u

(x), 1), − 1). (4)

The motivation behind this construction is to introduce linear

behavior around zero to allow gradients to ﬂow easily when

the unit is not saturated, while providing a crisp decision in

the saturated regime.

4 2 0 2 4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

derivative of hard-tanh

derivative of tanh

derivative of sigmoid

derivative of hard sigmoid

Figure 1. The plot for derivatives of different activation functions.

The ability of the hard-sigmoid and hard-tanh to make crisp

decisions comes at the cost of exactly 0 gradients in the

saturated regime. This can cause difﬁculties during training:

a small but not inﬁnitesimal change of the pre-activation

(before the nonlinearity) may help to reduce the objective

function, but this will not be reﬂected in the gradient.

In the rest of the document we will use h(x) to refer to

a generic activation function and use u(x) to denote its

linearization based on the ﬁrst-order Taylor expansion about

zero. hard-sigmoid saturates when x 6 −2 or x > 2 and

hard-tanh

saturates when

x 6 −1

x > 1

. We denote the

threshold by

. Absolute values of the threshold are

= 2

for hard-sigmoid and x

= 1 for the hard-tanh.

Remark. Let us note that both

hard-sigmoid(x)

sigmoid(x)

and tanh(x) are all contractive mapping. hard-tanh(x),

becomes a contractive mapping only when its input is greater

than the threshold. An important difference among these

activation functions is their ﬁxed points. hard-sigmoid(x)

has a ﬁxed point at x =

. However the ﬁxed-point of

sigmoid is x ≈ 0.69. Any x ∈ R between −1 and 1 can

be the ﬁxed-point of hard-tanh(x), but the ﬁxed-point of

tanh(x)

and

sigmoid(x)

have point attractors

at their ﬁxed-points. Those mathematical differences among

the saturating activation functions can make them behave

differently with RNNs and deep networks.

剩余9页未读，继续阅读

评论收藏

内容反馈

小小搬运工40

粉丝: 70

Noisy activation function

Tensorflow 9 激励函数 activation function (神经网络 教学教程tutorial)

Theano 7 activation function 激励函数 (神经网络 教学教程tutorial)

bp_v1_2.rar_activation function_matlab 数据归一化_激活函数_神经网络 线性函数_神经网络

Ollama软件windows安装包(版本0.3.10)

Page Assist - 本地 AI 模型(deepseek)的 Web UI

博客中聚类算法（K-means、FCM、DBSCAN、DPC）的数据集（免积分）

Chatbox-1.3.5-安装包

机器学习期末复习题及答案

神经网络回归预测--气温数据集

Ollama 0.5.7

DeepSeek R1 本地部署 桌面客户端 Windows版本

shape_predictor_68_face_landmarks.zip

Mathwork+Matlab+编程手册

人工智能导论（第5版）.pdf

影刀应用迁移工具，支持5.25最新版

Plecs电力电子仿真PLECS41.64 电力系统仿真软件免安装版本

时间序列预测模型实战案例(Xgboost)(Python)(机器学习)包括时间序列预测和时间序列分类，点击即可运行！

Claude window系统安装文件，有需要的可以自行下载

亚博K210模型训练部署

中文短信数据集-带标签

ollama 0.3.13

改进版的yolov5+双目测距

Ollama (windows版0.3.10)

mock_kaggle.csv

《1. 机器学习前置知识》配套数据集

电机故障数据集.rar

一款操作微信的RPA工具

OpenRPA安装包(Robot Framework、Taskt、UI.Vision、OpenRPA和TagUI)

基于鲸鱼优化算法优化VMD参数试看效果代码(目标函数为样本熵)

Stable-Diffusion WEBUI 简体中文语言包（2023.05.30更新）

高并发高流量网站架构

2013全新全天套黄发布站系统 Vs1.03

最新资源

Tensorflow 9 激励函数 activation function (神经网络教学教程tutorial)

Theano 7 activation function 激励函数 (神经网络教学教程tutorial)

bp_v1_2.rar_activation function_matlab 数据归一化_激活函数_神经网络线性函数_神经网络

DeepSeek R1 本地部署桌面客户端 Windows版本