[
exploratory
DSP
]
T
oday, signal processing
research has a significantly
widened its scope compared
with just a few years ago [4],
and machine learning has
been an important technical area of the
signal processing society. Since 2006,
deep learning—a new area of machine
learning research—has emerged [7],
impacting a wide range of signal and
information processing work within the
traditional and the new, widened scopes.
Various workshops, such as the 2009
ICML Workshop on Learning Feature
Hierarchies; the 2008 NIPS Deep
Learning Workshop: Foundations and
Future Directions; and the 2009 NIPS
Workshop on Deep Learning for Speech
Recognition and Related Applications as
well as an upcoming special issue on deep
learning for speech and language process-
ing in IEEE Transactions on Audio,
Speech, and Language Processing (2010)
have been devoted exclusively to deep
learning and its applications to classical
signal processing areas. We have also seen
the government sponsor research on deep
learning (e.g., the DARPA deep learning
program, available at http://www.darpa.
mil/ipto/solicit/baa/BAA-09-40_PIP.pdf).
The purpose of this article is to intro-
duce the readers to the emerging technol-
ogies enabled by deep learning and to
review the research work conducted in
this area that is of direct relevance to sig-
nal processing. We also point out, in our
view, the future research directions that
may attract interests of and require efforts
from more signal processing researchers
and practitioners in this emerging area
for advancing signal and information pro-
cessing technology and applications.
INTRODUCTION TO DEEP LEARNING
Many traditional machine learning and
signal processing techniques exploit shal-
low architectures, which contain a single
layer of nonlinear feature transformation.
Examples of shallow architectures are
conventional hidden Markov models
(HMMs), linear or nonlinear dynamical
systems, conditional random fields
(CRFs), maximum entropy (MaxEnt)
models, support vector machines (SVMs),
kernel regression, and multilayer percep-
tron (MLP) with a single hidden layer. A
property common to these shallow learn-
ing models is the simple architecture that
consists of only one layer responsible for
transforming the raw input signals or fea-
tures into a problem-specific feature
space, which may be unobservable. Take
the example of a support vector machine.
It is a shallow linear separation model
with one feature transformation layer
when kernel trick is used, and with zero
feature transformation layer when kernel
trick is not used.
Human information processing
mechanisms (e.g., vision and speech),
however, suggest the need of deep archi-
tectures for extracting complex structure
and building internal representation
from rich sensory inputs (e.g., natural
image and its motion, speech, and
music). For example, human speech pro-
duction and perception systems are both
equipped with clearly layered hierarchical
structures in transforming information
from the waveform level to the linguistic
level and vice versa. It is natural to
believe that the state of the art can be
advanced in processing these types of
media signals if efficient and effective
deep learning algorithms are developed.
Signal processing systems with deep
architectures are composed of many lay-
ers of nonlinear processing stages, where
each lower layer’s outputs are fed to its
immediate higher layer as the input. The
successful deep learning techniques
developed so far share two additional key
properties: the generative nature of the
model, which typically requires an addi-
tional top layer to perform the discrimi-
native task, and an unsupervised
pretraining step that makes effective use
of large amounts of unlabeled training
data for extracting structures and regular-
ities in the input features.
A BRIEF HISTORY
The concept of deep learning originated
from artificial neural network research.
Multilayer perceptron with many hidden
layers is a good example of the models
with deep architectures. Backpropagation,
invented in 1980s, has been a well-known
algorithm for learning the weights of
these networks. Unfortunately backpropa-
gation alone does not work well in prac-
tice for learning networks with more than
a small number of hidden layers (see a
review and interesting analysis in [1]).
The pervasive presence of local optima in
the nonconvex objective function of the
deep networks is the main source of diffi-
culty in learning. Backpropagation is
based on local gradient descent and starts
usually at some random initial points. It
often gets trapped in poor local optima
and the severity increases significantly as
the depth of the networks increases. This
difficulty is partially responsible for steer-
ing away most of the machine learning
and signal processing research from neu-
ral networks to shallow models that have
convex loss functions (e.g., SVMs, CRFs,
and MaxEnt models) for which global
optimum can be efficiently obtained at
the cost of less powerful models.
The optimization difficulty associated
with the deep models was empirically
IEEE SIGNAL PROCESSING MAGAZINE [145] JANUARY 2011
Digital Object Identifier 10.1109/MSP.2010.939038
1053-5888/11/$26.00©2011IEEE
Dong Yu and Li Deng
Deep Learning and Its Applications
to Signal and Information Processing
Date of publication: 17 December 2010
资源评论
Jason29zhang2014-01-08很有用啊，综述类的，刚开始接触的应该看看
jay7575
- 粉丝: 5
最新资源
- (源码)基于Python和Neo4j的智能就医系统.zip
- 监控专用网络EPON系统测试方案.doc
- Comsol与Matlab联合仿真及模型参数优化：以燃料电池流道优化为例
- 基于JAVA的餐饮管理系统毕业论文1.doc
- (源码)基于Arduino的生物机械手控制系统.zip
- 基于DSP的电机控制技术研究：无刷直流伺服电机的数学模型与控制策略实现
- COMSOL热-流-固三场耦合模拟煤层气藏注CO2开发及CCUS应用 - COMSOL 教程
- (源码)基于ROS的机器人感知与控制项目.zip
- 基于C#与西门子PLC的工控数据采集系统实战源码及精美UI ScottPlot 全面版
- (源码)基于Arduino UNO和TensorFlowKeras的MNIST手写数字快速分类系统.zip
- 三相PWM整流电路的双闭环控制与Simulink仿真实现及应用 - PWM调制
- (源码)基于Python和LightGBM的视频留存预测系统.zip
- 基于Carsim2020.0与Matlab Simulink2018b的7自由度车辆动力学模型联合仿真验证
- MATLAB Simulink中线性分组码BCH与卷积码的工程实现及误码率分析
- (源码)基于嵌入式C语言的LED矩阵贪吃蛇游戏.zip
- MATLAB频散曲线绘制软件：圆柱、圆环导波问题求解工具 - GUI界面
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
